fix(#546): CLI multi-word ranked search end-to-end (incl. first-cold-run) + audit bundle — callee comment stripping, codedb_query UAF, secret filters by justrach · Pull Request #572 · justrach/codedb

justrach · 2026-06-10T04:38:19Z

Problem

Two threads, rebased onto release/0.2.5825 and overlap-resolved against #557/#559/#561/#563/#565 (and re-rebased over #567/#571):

1. #546 — the CLI half. runQuery (cold CLI + warm cli-daemon) always called the unranked searchContent, so multi-word queries came back in recall order (usually nothing, since the literal phrase rarely appears); only the MCP handler ranked. Complements #557, which fixed the single-word rerank priors.

While validating live, two further cold-start defects surfaced — multi-word codedb search returned nothing on any cold start even with the routing fixed:

main.zig disables the word index for non-index commands, but word_index_complete defaults to true — the scan left an empty index claiming completeness, so searchContentRanked trusted the flag, skipped its lazy rebuild, and collapsed to N == 0.
The first-ever cold search scan (no trigram on disk) used the trigram-extract-only fast path, committing neither outlines nor contents — BM25 had nothing to rank even after a rebuild.

2. Audit (2026-06-09) latent-issue sweep — failing tests committed first for each:

codegraph.extractCallees paired every ( with the preceding identifier with no comment/string stripping → names mentioned in //, /* */, or string literals surfaced as resolved callees in codedb_context (complements fix(#562): codedb_callers excludes full-line comment mentions from call sites #563, same class in handleCallers)
codedb_query deps op appended freed dependency strings into file_set (use-after-free) and leaked the seed string
snapshot/watcher secret filter missed *.env suffixes (prod.env) and .git-credentials; fast-restore import read cap (4096) was narrower than the write cap (65535), silently disabling the borrow path
searchContent Tier-0 use_line_hits early-return skipped rerankAndFinalize (the search: structurally-relevant files rank below lexical hits (engram benchmark MRR 0.40 vs 0.57) #537/explore: rerank misses canonical files for basename-stem and case variants #448 inversion, reintroduced for small max_results); renderPlainSearch basename prior; >512KB evicted-file read cap

3. Appended: #568 + #569 (engram-bridge findings, red→green pairs):

deps: empty list prints '(none)' with no '(N files)' summary — inconsistent output corrupts machine consumers #568 — empty deps lists printed a body line (none) and no (N files) summary, so machine consumers parsing the list body saw one entry (engram counted in-degree 1 for every zero-importer file, degenerating its centrality features to all-1.0 ties).
search: multi-word queries return zero results — no per-token fallback in search or word #569 — multi-word word queries returned zero hits (literal-phrase lookup only; the multi-word analogue of codedb search returns no results for identifier terms that word indexes #547, which word didn't catch either). search multi-word is fixed by thread 1; this closes the word half.

Fix

Explorer.searchContentAuto — query-shape routing (multi-word → BM25+centrality, single token → literal) shared by runQuery and the MCP handler
commitParsedFileOwnedOutline clears word_index_complete when the disabled word index can't absorb a file, so ranked/BM25 readers lazy-rebuild instead of trusting a lying flag
first-cold multi-word non-regex searches route through the full single-pass scan (outlines + contents + trigrams); single-token and --regex keep the trigram-only fast path
comment/string spans stripped before call detection in extractCallees; codedb_query deps-op strings arena-owned; isSensitivePath extended + parity-kept; tier-0 results flow through rerank
deps: empty list prints '(none)' with no '(N files)' summary — inconsistent output corrupts machine consumers #568: (0 files) printed alongside (none) at all three deps render sites (imported-by fast path, general forward/transitive path, handleDepsPathOnly) — the summary line is now unconditional
search: multi-word queries return zero results — no per-token fallback in search or word #569: searchWordTokensLocked — whitespace-bearing queries tokenize (normalized, deduped), look up each token, and rank files by distinct-token coverage → total hits → path, one representative hit per file; shared by CLI searchWord and MCP renderWord (which labels the mode (tokenized))

Rebase notes

Replayed onto the release tip twice (51c3c39, then 43228cc after #567/#571 merged). Conflicts resolved keeping both sides: searchContentAuto threaded through #561's path_glob escalation loop (both fetch sites), all EOF tests preserved pairwise, #557's tooling-path prior verified intact in the merged rerankSignalScore chain (tests 0.6 → bench/scripts/website/install 0.5 → vendor 0.4 → doc cap).

Validation

zig build test: 749/749 pass (23/23 steps)
Live matrix on this repo, in-process (CODEDB_NO_CLI_DAEMON=1):
- first-ever cold multi-word search: 0 → 50 results (word index rebuild, snapshot load); one-time scan cost 722ms → 1.7s, trigram persists so later runs stay fast
- cold single-word (rebuildWordIndex): 9 results, fast path unchanged; cold --regex: 2 results, fast path unchanged
- warm (snapshot-loaded) multi-word: 50 results
- search: multi-word queries return zero results — no per-token fallback in search or word #569: codedb word "word index rebuild": 0 → 176 hits, files covering more tokens first
- deps: empty list prints '(none)' with no '(N files)' summary — inconsistent output corrupts machine consumers #568: codedb deps src/tests.zig (zero importers): now prints (none) + (0 files)
engram codedb_insights re-run post-fix: codedb MRR 0.398 vs engram re-rank 0.288 on the latest 25-commit window (the historical gap that spawned search: structurally-relevant files rank below lexical hits (engram benchmark MRR 0.36 vs 0.58) #546 was 0.30 vs 0.60 the other way)

🤖 Generated with Claude Code

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c79484ac6d

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-06-10T04:41:14Z

+            i += 1; // land on '/' of '*/'; the loop's i += 1 then moves past it
+            continue;
+        }
+        if (c == '"' or c == '\'') {


Avoid skipping Rust lifetimes as char literals

When analyzing Rust functions with lifetime annotations such as fn parse<'a>(...) { helper(); }, this branch treats the lifetime apostrophe as the start of a quoted literal and, if there is no matching ', advances to the end of the function body. That means real calls after the annotation are never extracted, dropping call graph edges/centrality/callpath for common Rust generic functions; the quote skipping needs to be language-aware or only apply to actual character literals.

Useful? React with 👍 / 👎.

github-actions · 2026-06-10T04:54:35Z

Benchmark Regression Report

Thresholds: 10.00% and 50,000 ns absolute delta

NOISE means the percentage threshold was exceeded, but the absolute delta was too small to fail CI.

Tool	Base (ns)	Head (ns)	Delta	Abs Delta (ns)	Status
`codedb_bundle`	108792	113575	+4.40%	+4783	OK
`codedb_changes`	11263	11483	+1.95%	+220	OK
`codedb_context`	1277949	1289758	+0.92%	+11809	OK
`codedb_deps`	352	346	-1.70%	-6	OK
`codedb_edit`	49255	52743	+7.08%	+3488	OK
`codedb_find`	10360	11733	+13.25%	+1373	NOISE
`codedb_hot`	29233	28334	-3.08%	-899	OK
`codedb_outline`	39378	37769	-4.09%	-1609	OK
`codedb_read`	20520	17637	-14.05%	-2883	OK
`codedb_search`	26590	28417	+6.87%	+1827	OK
`codedb_snapshot`	77723	79189	+1.89%	+1466	OK
`codedb_status`	9984	9860	-1.24%	-124	OK
`codedb_symbol`	53267	50184	-5.79%	-3083	OK
`codedb_tree`	30562	35360	+15.70%	+4798	NOISE
`codedb_word`	17150	11652	-32.06%	-5498	OK

github-actions · 2026-06-10T05:24:33Z

Benchmark Regression Report

Thresholds: 10.00% and 50,000 ns absolute delta

NOISE means the percentage threshold was exceeded, but the absolute delta was too small to fail CI.

Tool	Base (ns)	Head (ns)	Delta	Abs Delta (ns)	Status
`codedb_bundle`	102892	103490	+0.58%	+598	OK
`codedb_changes`	10650	10757	+1.00%	+107	OK
`codedb_context`	1203255	1206916	+0.30%	+3661	OK
`codedb_deps`	368	381	+3.53%	+13	OK
`codedb_edit`	44750	42381	-5.29%	-2369	OK
`codedb_find`	9668	9812	+1.49%	+144	OK
`codedb_hot`	24948	24724	-0.90%	-224	OK
`codedb_outline`	35614	34555	-2.97%	-1059	OK
`codedb_read`	16910	16399	-3.02%	-511	OK
`codedb_search`	26073	28790	+10.42%	+2717	NOISE
`codedb_snapshot`	62958	62346	-0.97%	-612	OK
`codedb_status`	9253	9420	+1.80%	+167	OK
`codedb_symbol`	48722	52162	+7.06%	+3440	OK
`codedb_tree`	28610	38144	+33.32%	+9534	NOISE
`codedb_word`	11869	11161	-5.97%	-708	OK

…ty, -22 LOC) Pure style: brace-wrapped single-statement for/while loops in 11 bool helpers collapsed to single-line form. No behavior change; test parity preserved. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…in searchContentRanked The engram MRR headline (0.36 vs 0.58) is a benchmark artifact (it models codedb as lexical-only), but it masked two real defects: 1. CLI `search` (runQuery — shared by the cold path and the warm cli-daemon) always called the UNRANKED searchContent, so multi-word queries came back in recall order; only the MCP handler routed multi-word queries to the BM25+centrality ranker. New Explorer.searchContentAuto centralises the query-shape decision (multi-word -> searchContentRanked, single token -> searchContent for exact-identifier lookups); runQuery and MCP handleSearch both call it, so CLI and MCP rank identically. 2. searchContentRanked was the only word-index reader that never lazily rebuilt an incomplete (mmap/snapshot-loaded) index — unlike searchContent, searchWord and renderWord. So even wired up, cold-CLI/daemon multi-word ranked search returned NOTHING: the index served `word` but BM25's N collapsed to 0. Added the same lazy rebuild (before the shared lock) so ranked search works for every caller. Before: `codedb . search "parse token"` -> no results. After: -> BM25+centrality-ranked hits (`search "search content"` ranks explore.zig #1). Tests (test_search.zig): searchContentAuto routes multi-word to the ranker; and searchContentRanked rebuilds an incomplete word index instead of returning empty (verified red by neutering the rebuild). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…xtraction extractCallees paired every ( with the preceding identifier with no comment or string stripping, so a name mentioned only in a // line comment, a /* block */ comment, or a string/char literal surfaced as a resolved callee in codedb_context. Strip those spans before call detection. Audit (2026-06-09) latent-issue sweep; failing tests added first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…e + leak) The deps op appended dependency strings into file_set that were freed each iteration alongside the per-file deps_result (use-after-free), and leaked the standalone seed-path dupe. Own both in a scoped arena freed at pipeline end. Audit (2026-06-09) latent-issue sweep; testing.allocator catches the poisoned read and the leak. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…fast-restore import cap Security: isSensitivePath (watcher.zig and snapshot.zig, parity-tested) missed *.env-suffix files such as production.env and staging.env, plus .git-credentials, so those secrets were indexed and read. Add the suffix match and exact name to both copies. Fast-restore: loadOutlineStateMap read OUTLINE_STATE imports with a 4096-byte cap while the writer allows u16 max, so a longer import silently disabled the borrow-path fast-restore. Widen the read cap to maxInt(u16). Audit (2026-06-09) latent-issue sweep; failing tests added first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Six latent issues in explore.zig: - parseDelimitedImport: strip a trailing `as <alias>` so Kotlin/Swift import X as Y keys on X, not the aliased form. - getImportedBy: skip the basename fallback when two or more indexed files share a basename, so a bare import conf is not attributed to every conf.py. - parsePythonLine: record one dep per comma-separated module in import a, b (was only the first); factor out appendPythonModuleDep. - searchContent tier0: drop the use_line_hits early-return that skipped rerankAndFinalize, so the canonical basename match outranks a high-count non-canonical file (the #537/#448 inversion at small max_results). - tier0 fast-path sort: add the path-prior (basename-stem/segment match plus test/vendor demotion) so renderPlainSearch and codedb_search no longer rank raw hit-count over the canonical file. - readContentForSearch: raise the disk-read cap 512KB to 64MB to match the indexer, so a word-indexed file larger than 512KB evicted from the content cache stays searchable. Audit (2026-06-09) latent-issue sweep; failing tests added first. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…mpty-but-complete word index; multi-word ranked search returns nothing Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Two cold-start defects made multi-word `codedb search` return nothing: 1. main.zig disables the word index for non-index commands, but word_index_complete defaults to true — committing files through a disabled index left an EMPTY index that claimed completeness, so searchContentRanked trusted the flag, skipped its lazy rebuild, and collapsed to N == 0. commitParsedFileOwnedOutline now clears word_index_complete when the disabled index can't absorb a file. 2. The first-ever cold `search` scan (no trigram on disk) used the trigram-extract-only fast path, committing neither outlines nor contents — BM25 had nothing to rank even after a rebuild. Multi-word non-regex searches now route through the full single-pass scan; single-token and --regex searches keep the trigram-only fast path. Live matrix on this repo: first-cold multi-word 0 -> 50 results (scan 722ms -> 1.7s, one-time; trigram persists so later runs stay fast); cold single-word/regex fast paths unchanged; warm paths unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…es)' summary Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…er-token fallback Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Empty lists printed a body line ' (none)' and no summary, so machine consumers that parse the list body saw one entry (engram counted in-degree 1 for every zero-importer file, degenerating its centrality features). Print '(0 files)' alongside '(none)' at all three render sites — the imported-by fast path, the general results path (forward/transitive), and handleDepsPathOnly — so the summary line is always present. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

'word' looked up the literal phrase in the inverted index, so an agent-shaped query like "gateway websocket reconnect" returned zero hits even when every token had plentiful hits. A whitespace-bearing query now routes through searchWordTokensLocked: each distinct (normalized, deduped) token is looked up and files are ranked by distinct-token coverage, then total hits, then path, returning one representative hit per file. Shared by the CLI (searchWord) and MCP codedb_word (renderWord, which labels the mode '(tokenized)'). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

github-actions · 2026-06-10T07:15:46Z

Benchmark Regression Report

Thresholds: 10.00% and 50,000 ns absolute delta

NOISE means the percentage threshold was exceeded, but the absolute delta was too small to fail CI.

Tool	Base (ns)	Head (ns)	Delta	Abs Delta (ns)	Status
`codedb_bundle`	106214	102747	-3.26%	-3467	OK
`codedb_changes`	10846	10368	-4.41%	-478	OK
`codedb_context`	1246571	1224751	-1.75%	-21820	OK
`codedb_deps`	444	303	-31.76%	-141	OK
`codedb_edit`	53122	42183	-20.59%	-10939	OK
`codedb_find`	9326	9331	+0.05%	+5	OK
`codedb_hot`	27289	24847	-8.95%	-2442	OK
`codedb_outline`	33367	35793	+7.27%	+2426	OK
`codedb_read`	15799	16355	+3.52%	+556	OK
`codedb_search`	28405	28470	+0.23%	+65	OK
`codedb_snapshot`	68620	66419	-3.21%	-2201	OK
`codedb_status`	9302	9232	-0.75%	-70	OK
`codedb_symbol`	50368	51695	+2.63%	+1327	OK
`codedb_tree`	39676	39445	-0.58%	-231	OK
`codedb_word`	11277	12098	+7.28%	+821	OK

…nd table Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

fix(#588): renderImportedBy honors the basename-ambiguity guard from #572

…empty symbol name not an error Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

…biguity, mirroring getImportedBy The justrach#572 bundle added the same-basename ambiguity guard to getImportedBy (query-pipeline deps op) but renderImportedBy — the MCP codedb_deps reverse listing — kept the unconditional fallback, so a bare 'import conf' was still attributed to every indexed conf.py through the primary tool. Count same-basename outlines and skip the fallback when ambiguous, exactly like getImportedBy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

chatgpt-codex-connector Bot reviewed Jun 10, 2026

View reviewed changes

justrach force-pushed the fix/issue-546-cli-ranked-search branch from c79484a to b4952f7 Compare June 10, 2026 04:51

justrach force-pushed the fix/issue-546-cli-ranked-search branch from 9d0f05d to 94fc30b Compare June 10, 2026 05:21

justrach and others added 12 commits June 10, 2026 15:09

test(#546): failing — cold CLI scan (word index disabled) leaves an e…

2bf2d68

…mpty-but-complete word index; multi-word ranked search returns nothing Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

test(#568): failing — deps empty list prints '(none)' with no '(N fil…

2408174

…es)' summary Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

test(#569): failing — multi-word 'word' query returns zero hits, no p…

3b577fd

…er-token fallback Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

justrach force-pushed the fix/issue-546-cli-ranked-search branch from 94fc30b to 7d6c537 Compare June 10, 2026 07:12

justrach merged commit de744e0 into release/0.2.5825 Jun 10, 2026
1 check passed

justrach deleted the fix/issue-546-cli-ranked-search branch June 10, 2026 07:13

justrach added a commit that referenced this pull request Jun 10, 2026

merge release/0.2.5825 (user's #572 audit bundle) — keep shared comma…

2d66639

…nd table Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This was referenced Jun 10, 2026

deps: renderImportedBy keeps the unconditional basename fallback — ambiguous bare imports still attributed to every same-basename file via codedb_deps #588

Closed

fix(#588): renderImportedBy honors the basename-ambiguity guard from #572 #590

Merged

justrach added a commit that referenced this pull request Jun 10, 2026

Merge pull request #590 from justrach/fix/issue-588-render-imported-by

491bbd9

fix(#588): renderImportedBy honors the basename-ambiguity guard from #572

justrach mentioned this pull request Jun 10, 2026

search: structurally-relevant files rank below lexical hits (engram benchmark MRR 0.36 vs 0.58) #546

Closed

idea404 pushed a commit to idea404/codedb that referenced this pull request Jun 10, 2026

test(justrach#572): failing — leading flag binds as positional name; …

4906f1e

…empty symbol name not an error Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

This was referenced Jun 11, 2026

deps: empty list prints '(none)' with no '(N files)' summary — inconsistent output corrupts machine consumers #568

Closed

search: multi-word queries return zero results — no per-token fallback in search or word #569

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix(#546): CLI multi-word ranked search end-to-end (incl. first-cold-run) + audit bundle — callee comment stripping, codedb_query UAF, secret filters#572

fix(#546): CLI multi-word ranked search end-to-end (incl. first-cold-run) + audit bundle — callee comment stripping, codedb_query UAF, secret filters#572
justrach merged 12 commits into
release/0.2.5825from
fix/issue-546-cli-ranked-search

justrach commented Jun 10, 2026 •

edited

Loading

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Jun 10, 2026

Uh oh!

github-actions Bot commented Jun 10, 2026

Uh oh!

github-actions Bot commented Jun 10, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jun 10, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

justrach commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Problem

Fix

Rebase notes

Validation

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Jun 10, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jun 10, 2026

Benchmark Regression Report

Uh oh!

github-actions Bot commented Jun 10, 2026

Benchmark Regression Report

Uh oh!

Uh oh!

github-actions Bot commented Jun 10, 2026

Benchmark Regression Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

justrach commented Jun 10, 2026 •

edited

Loading