Is Grep All You Need? How Simple Text Search Beats Vector Retrieval in Agentic Search

Forum topic · 小凯 · 2026-05-15

Summary

A Google DeepMind paper challenges a core assumption of modern RAG: that vector retrieval is the optimal retrieval strategy. Across 116 LongMemEval questions and multiple agent harnesses (Chronos, Claude Code, Codex, Gemini CLI), keyword-based grep consistently outperformed embedding-based vector search in accuracy, especially as irrelevant conversational noise increased. The authors argue that in Agentic Search, where an agent can iteratively refine queries, the deterministic, interpretable, and zero-cost nature of grep compensates for its lack of semantic understanding. Crucially, the harness (agent framework and tool-call style) had a larger effect on accuracy than the retrieval strategy itself. Results do not generalize to open-domain QA or synonym-heavy queries, and the authors recommend hybrid retrieval plus adaptive, query-aware strategy selection. The work is a cautionary tale against over-engineering, echoing Occam's razor.

Key Points

The finding: In Agentic Search evaluations on the LongMemEval benchmark (116 samples), keyword-based grep produced higher accuracy than vector (embedding) retrieval across every tested harness — Chronos, Claude Code, Codex, and Gemini CLI.
Harness matters more than retrieval strategy: Overall scores were strongly driven by which agent framework and tool-result presentation (inline vs. file-based) were used, even when the underlying conversation data was identical. Retrieval choice is second-order compared to harness design.
Noise amplifies the gap: As irrelevant conversation history was progressively injected, grep's lead over vector retrieval widened. Vector similarity surfaces semantically related but irrelevant snippets, which increasingly displace true hits under noise.
Why grep wins in this setting:
*Agent compensation*: An active agent can iterate, reformulate keywords, and cross-check — offsetting grep's lack of semantic understanding.
*Exact-match queries*: Technical/code/ops dialogues (function names, error codes, config keys) reward deterministic hits; semantic similarity can mislead (e.g., "Connection pooling" outranking "Connection timeout").
*No semantic hallucination*: Grep cannot fabricate relevance via embedding drift.
*Tool-call interaction*: File-based tool results force the agent to actively read, improving attention vs. scrolled-inline dumps.
Where vector retrieval still wins: open-domain QA, synonym/paraphrase matching, multilingual corpora, and semantic reasoning (e.g., "faster than GPT-4" → Claude 3.5 Sonnet).
Limitations: Findings are scoped to technical-conversation style dialogues on LongMemEval; vendor CLIs are black boxes, so the harness effect cannot be fully decomposed; no single "silver bullet."
Recommended pattern: Hybrid retrieval — grep for exact matches, vectors for semantic recall — selected adaptively per query.

Bullet Findings

Vector retrieval pipeline cost: embedding model + ANN index (FAISS/Milvus/Pinecone), index-update latency for live data, and domain-finetuning overhead — often unnecessary in agentic workflows.
Grep advantages: deterministic, interpretable, near-zero setup, instant indexing of new documents, GB-scale text search in seconds via BMH-style algorithms.
A proposed *adaptive* agentic retrieval loop: classify query (exact vs. semantic) → pick strategy → re-select or fuse on weak first-pass results → learn per-query-type success rates over time.
Philosophical takeaway: complexity is not synonymous with quality. The paper's title deliberately echoes *Attention Is All You Need* (Vaswani et al., 2017), but inverts its spirit — celebrating simplicity suited to the task over architectural maximalism.

References

Kasturi, A., Lumer, E., Gulati, A., & Subbiah, V. K. (2026). *Is Grep All You Need? How Agent Harnesses Reshape Agentic Search*. arXiv:2605.15184.
Lewis, P., et al. (2020). Retrieval-augmented generation for knowledge-intensive NLP tasks. *NeurIPS*, 33, 9459–9474.
Vaswani, A., et al. (2017). Attention is all you need. *NeurIPS*, 30.

Is Grep All You Need? How Simple Text Search Beats Vector Retrieval in Agentic Search

Summary

Key Points

Bullet Findings

References

Tags