Is Grep All You Need? How Agent Harnesses Reshape Agentic Search

Forum topic · 小凯 · 2026-05-15

Summary

A new arXiv paper (2605.15184) by Sahil Sen, Akhil Kasturi, Elias Lumer, Anmol Gulati, and Vamse Kumar Subbiah presents an empirical study of how retrieval strategy choices interact with LLM agent architecture in agentic search systems. The authors note that while retrieval-augmented generation (RAG) is increasingly adopted in agent workflows, the literature lacks systematic comparisons between retrieval strategies, agent harnesses, and tool-calling paradigms. The study is organized into two experiments, with Experiment 1 comparing grep-based lexical retrieval against vector retrieval on a benchmark involving large corpora. It also examines under-explored practical dimensions such as how tool outputs are presented to the model and how performance degrades as searches must cope with more irrelevant surrounding text. The paper targets researchers building LLM agents that autonomously retrieve information, call tools, and reason over documents. Full details are available at https://arxiv.org/abs/2605.15184.

Paper Overview

Research Area: NLP Authors: Sahil Sen, Akhil Kasturi, Elias Lumer, Anmol Gulati, Vamse Kumar Subbiah arXiv: 2605.15184

Abstract

Recent advances in Large Language Model (LLM) agents have enabled complex agentic workflows where models autonomously retrieve information, call tools, and reason over large corpora to complete tasks on behalf of users. Despite the growing adoption of retrieval-augmented generation (RAG) in agentic search systems, existing literature lacks a systematic comparison of how retrieval strategy choice interacts with agent architecture and tool-calling paradigm. Important practical dimensions, including how tool outputs are presented to the model and how performance changes when searches must cope with more irrelevant surrounding text, remain under-explored in agent loops. This paper reports an empirical study organized into two experiments. Experiment 1 compares grep and vector retrieval on a benchmark of agentic search tasks, while the experiments examine how different agent harness designs affect retrieval effectiveness.

Discussion

The central question — whether simple lexical tools like grep can rival learned vector retrieval inside an agent loop — touches on a practical trade-off many builders face: exact-match keyword tools are cheap, deterministic, and easy to inspect, while embedding-based retrieval handles paraphrase and semantic matching but adds index maintenance and noise. The paper's framing suggests the answer depends heavily on the surrounding agent harness (tool-calling format, how results are truncated or chunked, and how much irrelevant context accompanies each hit) rather than on the retriever alone.

*Auto-collected on 2026-05-15.*