Is Grep All You Need? When a 1974 Tool Outperforms Vector Retrieval in Agentic Search

Forum topic · 小凯 · 2026-05-15

Summary

A Google DeepMind paper titled 'Is Grep All You Need? How Agent Harnesses Reshape Agentic Search' (arXiv:2605.15184) reports that simple keyword-based grep search consistently outperforms vector retrieval in agentic search setups. Experiments on 116 LongMemEval samples across four agent harnesses (Chronos, Claude Code, Codex, Gemini CLI) found grep accuracy generally beat embedding-based vector search, with the gap widening as irrelevant conversation history (noise) increased. The paper also shows overall scores depend more on the agent harness and tool-calling style (inline vs file-based results) than on the retrieval strategy itself. Plausible explanations include agent iterativeness compensating for grep's lack of semantics, exact-match suitability for technical queries (function names, error codes), and vector search's tendency to retrieve semantically similar but irrelevant chunks. The authors caution that LongMemEval covers technical dialogue mainly, that grep cannot handle synonyms, multilingual queries, or semantic inference, and recommend hybrid retrieval and query-adaptive strategy selection rather than abandoning vector search outright.

A recent Google DeepMind paper — *Is Grep All You Need? How Agent Harnesses Reshape Agentic Search* (arXiv:2605.15184) — makes a counterintuitive claim: in Agentic Search, simple keyword grep often achieves higher accuracy than vector retrieval, the de facto standard of modern RAG pipelines.

Background: Vector Retrieval vs. grep

Vector retrieval in RAG works by embedding text chunks into 768- or 1536-dimensional vectors (via models like BERT or OpenAI's text-embedding models), indexing them in vector databases (FAISS, Milvus, Pinecone), and performing approximate nearest-neighbor search at query time. Its advantages: semantic understanding, millisecond-scale queries over huge corpora, and generality. But it carries costs:

Semantic drift: compression into fixed-dimension vectors loses detail, hurting exact matching (function names, error codes).
Indexing overhead: every new document needs embedding and index updates.
Black-box behavior: irrelevant results are hard to explain.
Domain adaptation: general embedding models underperform in specialized domains.

grep, the 1974 Unix text-search tool, is deterministic, explainable, fast, and zero-cost (no embeddings, no index — new documents are instantly searchable). But it cannot handle synonyms, multilingual queries, or semantic inference.

Experiments

Setup: 116 samples from LongMemEval, tested with two retrieval strategies (grep vs vector) and two tool-result presentation styles (inline vs file-based) across four agent harnesses: Chronos (the authors' custom framework), Claude Code, Codex, and Gemini CLI.

A second experiment progressively mixed in more irrelevant conversation history to test robustness under noise.

Key Findings

1. grep beats vector retrieval across all harnesses — a consistent trend regardless of framework. 2. The harness matters more than the retrieval strategy: scores depend strongly on which harness and tool-calling style is used, even with identical underlying data. 3. Noise widens the gap: as irrelevant history grows, grep's deterministic matching stays precise, while vector search's semantic similarity increasingly surfaces irrelevant-but-lookalike chunks. 4. Tool-result presentation (inline vs. file-based) significantly affects outcomes, showing retrieval interacts with the agent's overall workflow.

Why grep Wins in Agentic Search

Agent agency compensates for grep's 'dumbness': unlike passive one-shot RAG, an agent can search repeatedly, refine keywords, and cross-validate.
Exact matching suits technical queries: e.g., searching "Connection timeout" correctly finds "Error: Connection timeout after 30s" rather than semantically similar but irrelevant "Connection pooling configuration".
No retrieval 'hallucinations': grep never returns semantically similar but wrong chunks.

Caveats

LongMemEval mainly covers technical dialogue; grep's advantage may not extend to open-domain QA or other domains.
grep fails on synonyms, cross-lingual queries, and semantic reasoning — vector retrieval remains essential there.
Vendor CLIs are black boxes, making the harness effect hard to fully explain.
The conclusion is not 'replace vector search with grep' but that retrieval strategy should be chosen jointly with agent architecture, task type, and tool-calling paradigm — hybrid retrieval (grep for exact match, vectors for semantics) is likely optimal.

Takeaway

The paper's real contribution is opening an overlooked direction: retrieval strategy selection should be adaptive — agents analyzing query type, choosing grep or vector search accordingly, and switching dynamically. In an industry driven by 'bigger, stronger, more complex,' it's a reminder that a 1974 tool still has an irreplaceable place in 2026 AI systems — not because technology hasn't advanced, but because a technique's value depends on the scenario.

References

Kasturi, A., Lumer, E., Gulati, A., & Subbiah, V. K. (2026). *Is Grep All You Need? How Agent Harnesses Reshape Agentic Search*. arXiv:2605.15184.
Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. *Advances in Neural Information Processing Systems*, 33, 9459-9474.
Vaswani, A., et al. (2017). Attention is all you need. *Advances in Neural Information Processing Systems*, 30.