Why Claude Code Ditched RAG for Agentic Search

Background: The Problem with Traditional RAG in Large Codebases

Imagine a senior engineer dropped into a project with millions of lines of code, given a bug to fix, and handed a thick "code encyclopedia" printed three months ago—covering only five pages at a time. That, the article argues, is a good metaphor for traditional RAG (Retrieval-Augmented Generation) in large codebases.

Anthropic's Claude Code team initially built such a system using the Voyage vector database, embedding the entire codebase into high-dimensional vectors. It worked well on small projects, but as the codebase grew to millions of lines with dozens of commits per hour, the system began to fail:

• Stale indexes: Vector databases are static; re-embedding after every change can take hours or days. Outdated vectors led the model to reference deprecated APIs and deleted functions.
• Security risks: Vectorized source code—containing trade secrets and unreleased algorithms—had to be stored somewhere (third-party services or internal databases), creating an unacceptable attack surface for Anthropic.
• Engineering complexity: Maintaining embedding pipelines, index updates, monitoring, and recovery consumed nearly as much effort as building core features, and debugging poor retrieval results was like groping inside a black box.

The Shift: Agentic Search

Core developer Boris Cherny (speaking on the Latent Space podcast) proposed a counterintuitive idea: instead of making the AI memorize everything, teach it to look things up itself. Agentic Search lets the model perform multi-round, proactive searches just like a developer using everyday tools:

• grep — search strings/regex in files
• glob — pattern-based file lookup (e.g., src/**/*.ts)
• find definition / find references
• git log — change history
• cat, ls — read files, list directories

When asked "how is user authentication implemented?", the model doesn't passively receive a few retrieved files. It might: (1) glob for auth-related files, (2) cat key files, (3) find definition on a discovered validate_token function, (4) find references to understand usage, (5) git log to check recent changes. The result is built on live, accurate, complete information rather than an outdated vector snapshot.

Why It Works Better

From intuition to evidence. The team initially adopted Agentic Search on intuition ("vibes")—it felt closer to how developers think. After long-term practice, they found code generation quality was higher and responses more natural. Internal testing reportedly showed it "significantly outperformed all other approaches," improving accuracy (no reliance on stale APIs), relevance (iterative narrowing excludes noise), completeness (find references catches all call sites), and timeliness (always operating on the latest code).

Security. Boris Cherny noted that vector data "must be hosted somewhere—third-party or internal—and both face leakage risks." Agentic Search requires no external storage: searches execute directly on the live system through standard, OS-permission-controlled tool calls. Every step is explicit, loggable, and auditable by existing SIEM systems—a major plus for regulated industries. Real-time execution also eliminates the dangerous synchronization window where RAG serves outdated information.

Developer experience. Retrieval behavior is transparent and reproducible: developers can see and verify each step (which files were searched, which functions examined). Using familiar tools like grep and git means near-zero learning curve, easy debugging, and natural extensibility. Because the model reads real project code, generated code matches project conventions—naming, error handling, architecture patterns—hence the "more natural" responses.

The Trade-offs

Cherny is candid: "at the cost of latency and token expense, we gain stronger, safer search capability."

• Latency: RAG retrieval completes in hundreds of milliseconds; an Agentic Search session may involve 5–10 tool calls plus re-reasoning between them, taking roughly 5–15 seconds. Unacceptable for autocomplete, but acceptable for code generation, review, and architecture tasks.
• Token cost: Each tool call consumes tokens (descriptions, results, conversation history); a complex session may cost tens of thousands of tokens—dozens of times a single RAG retrieval.
• Mitigations: result caching, parallel independent calls, early termination once sufficient information is gathered, and tiered strategies (lightweight tools for simple queries, full agentic search for complex tasks).

A Paradigm Shift—and Its Limits

The article frames this as a move from "open-book exam" (the proctor decides which pages you see) to "survival in the wild" (you explore with basic tools). AI's capability boundary expands from context-window size to the richness of its toolset and exploration strategy.

But Cherny cautions that code is highly structured—clear hierarchy, precise identifiers, strict syntax—which makes tool-based search natural. Unstructured long documents (legal contracts, financial reports, papers) are far messier: hundreds of pages, no clear markers, scattered information.

PageIndex: Extending the Idea to Long Documents

PageIndex applies the agentic-retrieval philosophy to long documents as the first In-Context Index system for LLMs. Instead of embedding documents into a vector database, it builds a hierarchical, table-of-contents-style tree index and places the entire tree inside the model's context window. Retrieval becomes: navigate the tree → reason about relevant sections → locate → read the actual text → integrate, optionally looping back to the index.

In financial document QA—where an IPO prospectus can run 800 pages—a PageIndex-based system achieved state-of-the-art results without any vector database, answering questions like R&D spending as a share of revenue by navigating to the income statement, extracting figures, and computing the ratio, with fully traceable reasoning.

Conclusion: Memory vs. Intelligence

Traditional RAG tries to hand the AI a perfect encyclopedia; Agentic Search gives it a compass and sextant. The deeper question: should AI memorize the world, or learn to explore it? Anthropic chose exploration—trading speed and cost for accuracy, security, and explainability. As Cherny put it: "We tried several different retrieval methods and search tools, and concluded that Agentic Search is the better solution." The lesson for practitioners: even widely adopted "standard answers" like RAG can fail in specific contexts, and real innovation often comes from returning to first principles.

References

1. Cherny, B. (2024). *Claude Code: Anthropic's CLI Agent*. Latent Space Podcast interview. 2. OfficeChai (2024). *Claude Researcher Explains How Agentic Search Performed Better Than RAG for Code Generation*. 3. Lewis, P., et al. (2020). "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks". *NeurIPS*, 33, 9459–9474. 4. VoyageAI (2023). "Vector Embeddings for Code Search: Challenges and Best Practices". 5. PageIndex Research Team (2024). "In-Context Indexing for Long Document Analysis".