My context selector beat grep. An agent with grep beat it.

cognitive-cache started from an observation I still think is correct: every coding tool decides what goes into the context window, and almost none of them treat that decision as an explicit selection problem. Cursor greps, RAG systems embed and cosine-search, and plenty of harnesses just stuff files until the budget runs out. The OS analogy is hard to unsee once you have it. The context window is RAM, eviction and retrieval are virtual memory, and token budget allocation is an allocator that nobody has written.

So I wrote one. Six weighted signals score every file in a repo against a task description, and a two-phase greedy submodular selector picks the highest-value set that fits a token budget. Nine languages, no LLM calls, no API keys. It shipped to PyPI and I used it through MCP for a while.

Then I built a benchmark harness, and the harness spent the next two months dismantling most of what I believed about the tool. This is that sequence, in order, because the order is the interesting part.

The first honest measurement said it does not beat grep

The original benchmark was bad in the way most self-benchmarks are bad. It coupled retrieval to LLM patch generation, reported rank-blind full-set recall, had no significance test, and I almost never ran it.

The replacement is LLM-free and deterministic. For each issue it clones the repo at the pre-fix commit, produces a full file ranking per strategy, and scores that ranking against the files the gold patch actually modified, using recall@k and MRR. Every head-to-head comparison carries a paired bootstrap 95% confidence interval, so a difference that excludes zero is real and a difference that straddles zero is not a result.

On the first dataset, 23 hand-curated issues across 8 repos:

The recall@5 difference over grep was +0.020 with a 95% CI of [-0.067, +0.109]. Grep won at recall@10. Whatever the tool was doing, at that sample size it was indistinguishable from counting keywords.

Two other things fell out of the same run. The per-signal ablation suggested graph_distance, carrying 0.20 of the weight budget, was net negative, and file_role_prior was doing nothing at 0.07. And the budget selector was leaking badly: ranking recall@5 was 0.428 while end-to-end recall under a 12k budget was 0.282, so the selector was discarding relevant files its own ranking had already surfaced.

I nearly acted on the ablation immediately. A weight sweep stopped me, because every candidate reweighting had a bootstrap lower bound of +0.000. The "graph_distance is net negative" reading was itself noise at n=23. The real blocker was not the weights, it was that 23 narrow issues cannot detect or drive an improvement in either direction.

Widening the dataset flipped the result

I moved to SWE-bench, which solves the curation problem by construction: real GitHub issues paired with the gold patch that fixed them, pulled over the HuggingFace datasets-server with no auth. Ground truth is the set of source files the patch modified, and because patches only touch files present at the base commit, every ground-truth file is retrievable.

On a balanced 78-instance subset of SWE-bench Lite across 12 repos:

Recall@5 over grep: +0.179, CI [+0.064, +0.295]. Over TF-IDF: +0.128, CI [+0.013, +0.244]. Both exclude zero. On a harder 91-issue subset restricted to fixes touching two or more files, the edge held at 0.365 vs 0.252, CI [+0.056, +0.174].

The earlier negative was a small-sample artifact. That cuts both ways, and it is worth sitting with: if I had published the n=23 result I would have been wrong, and if I had published the n=78 result as the final word I would also have been wrong, for the reason in the next section.

The baseline I had not run was an agent

"Beats grep" means beats a naive keyword count. It says nothing about the thing my tool would actually be competing with, which is a coding agent running its own search loop.

So I ran that. Five single-file SWE-bench issues, one each from flask, requests, xarray, pylint, and pytest. Fresh Claude subagents, cold in the repo at the base commit, grep, glob, and read only, cognitive-cache explicitly forbidden.

The agent got 5 of 5, averaging about 3.4 tool calls and 19k tokens, and in each case it pinpointed the mechanism rather than just the file. cognitive-cache got 4 of 5, missing pylint #5859, which the agent found in two calls. Naive keyword grep got 3 of 5.

The token argument inverted at the same time. The agent read a handful of targeted files. A cognitive-cache call returns file contents up to the full token budget, so on these tasks it front-loads more context than a surgical agent consumes. It was costing context, not saving it.

That is n=5 on hand-picked single-file issues, so it is directional rather than definitive. But it moved the honest positioning from "more accurate than the agent" to "fewer roundtrips on a cold start in unfamiliar code", which is a much smaller claim.

The signal that was worth zero, and why

graph_distance measures import-graph distance from the files the task names. It carried 0.20 weight from the day I wrote it, on the reasoning that a fix often lives one hop away from the file the issue mentions.

Across both datasets it earned nothing. On the 91-issue multi-file set, zeroing it changed recall@5 by +0.000 and MRR by -0.006. That was the pre-registered kill criterion, so I was ready to retire it.

Then I pushed back on my own experiment, which turned out to be the useful move. Multi-file is not the same as multi-hop, a regex-based import graph is not a reference graph, and testing a signal as one term in a drowned weighted sum is not a fair test of the underlying idea. All three critiques were fair, so instead of assuming, I wrote a diagnostic over the 91-issue set to check whether the ground truth was even reachable.

The answer killed the revival rather than supporting it. 93% of issues have all their ground-truth files reachable from the entry points. But 87% of fix files are themselves entry points, at graph distance 0, with only about 10% sitting at one to three hops and none further out. The mechanism was never starvation, it was redundancy: real issue text names the files it touches, so symbol_overlap has already ranked those files highly and import distance adds nothing on top.

That is not a bug in the experiment. It is a fact about the task distribution, and it explains the agent result too. On issue-shaped work the target is usually named in the prompt, which is exactly the regime where an agentic grep loop is excellent and a static ranker has little room to add value. Claude Code shipping no code index looks less like an omission and more like the same finding, reached earlier.

The delivery problem, measured

One thread survived long enough to be worth reporting separately. I built a jedi-backed neighborhood tool (definition, callers, in-repo callees, with real scope-aware resolution rather than name matching) and tested it two ways.

Against agents, on two hard cross-file issues where naive grep recall was 0, I ran a 2x2: agents with grep and read, versus agents offered the neighborhood tool as well. Every arm scored 0.5, finding the file tied to the named concept and missing the cross-file one. The agents that had the tool invoked it zero times out of two, despite being told to. Checking the tool's output directly against the symbols those same agents had reasoned about, it surfaced both missed files. The data would have lifted recall from 0.5 to 1.0 if it had been used.

At scale the value is real but modest: across all 91 multi-file issues, neighborhood-of-seeds alone scores 0.164 (worse than grep's 0.252 standalone), while grep-or-neighborhood combined reaches 0.330, recovering 20 of grep's 257 missed fix files. That is +7.8 absolute points, in line with RepoGraph's own reported 9-12%.

Put the two halves together and the conclusion is about delivery, not capability. A passive tool that competes with the grep reflex for invocation loses the invocation. Value like that only lands if it is injected into results the agent is already reading, not offered as one more thing it could call.

Where it ended

I set graph_distance to weight 0, rewrote the README around the scoreboard instead of the pitch, and stopped building. The remaining pitch would have required a forced-injection layer to capture a modest gain, in a niche that already has direct clones, running against the current of agents that search well on their own.

What survives is the measurement rig: the SWE-bench importer that needs no token, the retrieval evaluator, the weight tuner, the multi-hop diagnostic, the neighbor-recovery harness, and the bootstrap CIs that make each of those trustworthy. In a category that ships on anecdotes, that is the part I would build again first.

The generalizable lesson is not "context selection is a dead end". It is that the baseline you skip is the one that decides whether your work matters. I measured against grep for two months because grep was the baseline that let the tool win. The agent was two hours of work and it reframed the entire project.