When agents search: grep, vector retrieval, and why the harness decides
In a direct comparison over long chat histories, plain grep beat vector search for every model and harness when tool results came back inline — but moving the same agent between Claude Code, Codex, Gemini CLI, and a custom harness shifted accuracy as much as swapping the retriever did.
The problem
An agent that answers questions over a large corpus must search, because the corpus does not fit in its context window. Builders face two standard options. Vector search embeds the query and the documents into a shared numeric space and returns the nearest neighbors. Lexical search — grep with regular expressions, or a keyword ranker such as BM25 — matches literal strings in the raw text. Vector search is the default in most retrieval-augmented generation (RAG) stacks. Grep needs no embedding model, no index, and no infrastructure.
Existing benchmarks compare these retrievers in a fixed pipeline: run one query, paste the top results into a prompt. Agents do not work that way. An agent picks its own search terms, runs several queries, inspects the results, and decides when to stop. The agent harness — the loop that builds the prompt, dispatches tool calls, and formats results — mediates every step. The authors ask a question nobody had measured: does the grep-versus-vector result hold inside real harnesses?
The setup
The team tested 116 questions from LongMemEval, a benchmark where an agent must answer questions about long, multi-session chat histories. Each question comes with sessions that contain the answer and many irrelevant distractor sessions that do not. The corpus for each question holds the raw conversation turns plus structured records of dates and events, extracted in advance by the authors' Chronos preprocessing pipeline.
They varied three dimensions. First, the retriever: a grep tool that runs regular expressions over local files, against a vector tool with an embedding index and a reranker — a second model that re-orders the results by relevance. Second, the harness: their own custom agent (Chronos, built on LangChain, with per-question-type prompts) against three provider CLI agents — Claude Code, Codex, and Gemini CLI — which call the same tools through shell scripts. Third, the delivery mode: inline, where search results enter the conversation directly, against programmatic, where results go to a file that the agent must open with a second tool call. They ran five models, from Claude Opus 4.6 and GPT-5.4 down to Gemini 3.1 Flash-Lite; GPT-4o graded the answers.
Results
With inline delivery, grep won all ten harness–model pairings. The margin was sometimes large (86.2% against 62.9% for Gemini 3.1 Flash-Lite on the custom harness) and sometimes small (76.7% against 75.0% for Claude Opus 4.6 on Claude Code). The authors' explanation: these questions reward the recovery of literal strings — dates, counts, stated preferences. Grep returns exactly those strings. Vector search returns semantically near passages, which include on-topic distractors.
The second finding is the one that changes how you should read every agent benchmark. The same Claude Opus 4.6 model scored 93.1% inside the custom harness and 76.7% inside Claude Code, on identical data. Changing the harness moved accuracy as much as changing the retriever. The harness controls the system prompt, the tool descriptions, and how results render into the transcript — the authors argue you never measure "retrieval" alone, you measure retrieval plus orchestration.
Delivery mode was just as disruptive. Under file-based delivery, vector search overtook grep on five of ten pairings, and one configuration failed badly: Codex with GPT-5.4 fell from 93.1% with inline grep to 55.2% with file-based grep. Writing results to a file converts one read step into a locate-open-integrate workflow. When the agent executes that workflow unreliably, accuracy falls regardless of how good the retrieval was.
The second experiment increased the noise: it grew the number of irrelevant sessions around the answer from 5 up to the full set of 39–66. A common assumption says lexical search works on small corpora and semantic search becomes necessary at scale. The data agrees only in part. Vector search was often stronger at small session counts, but the crossover point depended on the harness and the model, not on corpus size alone. Some pairings showed a stable direction across the whole sweep: Claude Code favored grep with both Claude models at every size, while Gemini CLI favored vector search with Gemini 3.1 Pro at every size. The authors read these stable, vendor-specific patterns as built-in tendencies of each CLI's tooling — the default hints, the way stdout enters the transcript, the phrasing the agent uses for searches.
Caveats
The authors state the limits clearly, and the limits matter. The benchmark is one task family: conversational memory, where the answer appears word-for-word in the text — the case where lexical matching is strongest. The authors state directly that they do not claim grep wins in general; paraphrase-heavy or code-semantics corpora may favor vector search. The sample is 116 questions, graded by an LLM. Some Codex rows in the scaling experiment are incomplete. And the noise-sweep curves are not smooth laws: distractors are resampled at each size, so a peak at a single point can come from that sampling, not from a real effect.
What you can do with it
First: do not pick a vector database by default. If your agent searches data where answers appear as literal strings — logs, configuration, chat history, structured records — grep over local files is a strong baseline with zero infrastructure. Measure before you add an embedding pipeline.
Second: preprocessing can matter more than retriever choice. The authors' corpus worked for grep because dates and events were extracted into clean, matchable records first. Structure your data so that literal search can succeed.
Third: benchmark retrieval inside your actual harness, not in a standalone pipeline. A retriever comparison made in one harness did not transfer to another in this study, and a model that switches CLI stacks does not keep its retrieval behavior even on byte-identical data.
Fourth: treat file-based tool output as a trade-off, not an upgrade. It saves context-window space, but it adds a multi-step read workflow that weaker configurations fail. Adopt it only where you have verified the agent completes that loop.