Why Claude Code Ditched RAG for Agentic Search: Letting the Model Investigate Code Itself

Forum topic · QianXun · 2025-11-09

Summary

The Claude Code team at Anthropic abandoned traditional RAG (retrieval-augmented generation) in favor of Agentic Search for code generation. Core developer Boris Cherny, speaking on the Latent Space podcast, explained that their earlier Voyage vector-database approach degraded as codebases grew to millions of lines with frequent commits: vector indexes lagged behind code changes, retrieval results became stale and misleading, and maintaining embedding infrastructure proved costly. Storing code as vectors also created data-leakage risks for sensitive proprietary code. Agentic Search instead lets the model actively investigate the live repository using familiar developer tools such as grep, glob, find definition, find references, git log, and cat, performing multi-round, iterative searches that mirror how human engineers explore code. The team found this approach significantly outperformed RAG in accuracy, completeness, timeliness, and natural code style, at the cost of higher latency and token consumption, which can be mitigated through caching, parallel calls, early termination, and tiered strategies. The article also discusses extending agentic retrieval to unstructured long documents via PageIndex, a hierarchical in-context tree index that achieved state-of-the-results in financial document QA without any vector database, signaling a broader shift from memory-centric to reasoning-centric AI retrieval.

Key points

Anthropic's Claude Code team initially built a conventional RAG pipeline for code search using Voyage vector embeddings, but ultimately replaced it with Agentic Search, in which the model actively queries the codebase using standard developer tools.

Why traditional RAG fell short

Staleness: Vector indexes must be re-embedded after every commit. In large, fast-moving repositories, the index lags hours or days behind, so the model retrieves outdated functions, deprecated APIs, and misleading context.
Security: Embeddings must be stored somewhere—third-party services (Pinecone, Weaviate) or internal systems—creating an extra attack surface for proprietary code. Cherny: vector data must be hosted somewhere and either way faces leakage or attack risk.
Engineering burden: Embedding services, index pipelines, update scheduling, monitoring, and opaque black-box debugging consumed significant maintenance effort.
Opacity: RAG gives no explanation of why certain chunks were retrieved or whether they reflect the latest code.

How Agentic Search works

Instead of passively receiving retrieved snippets, the model acts like a detective at the scene, chaining tool calls such as:

grep — search strings/regex in files
glob — pattern-based file lookup (e.g., src/**/*.ts)
find definition / find references — semantic navigation
git log — change history
cat / ls — read files and list directories

A typical flow: glob for auth-related files → read key files → locate validate_token → inspect its references → check git history → synthesize an answer. This iterative, multi-round process builds understanding from live, accurate, complete information rather than a stale vector snapshot.

Measured and felt benefits

Initially adopted on intuition ("vibes"); after long-term use, the team found code generation quality noticeably higher and responses more natural.
Advantages span accuracy (no deprecated-API citations), relevance (context disambiguates user_interface.js vs user_authentication.ts), completeness (find references catches all call sites in refactors), and timeliness (always operating on the latest code).
Security: no external vector store; all access stays inside the existing OS permission boundary and produces standard, SIEM-friendly audit logs.
Developer experience: every retrieval step is explicit, reproducible, and verifiable; the model absorbs project conventions directly from real code, yielding idiomatic output.

The trade-offs

Latency: RAG retrieval is sub-second; agentic sessions involve 5–10+ tool calls and can take 5–15 seconds. Unacceptable for autocomplete, acceptable for code generation and review.
Token cost: tens of thousands of tokens per complex session—dozens of times a single RAG retrieval. Mitigations include result caching, parallel tool calls, early termination, and tiered strategies (lightweight tools for simple queries).

Extending beyond code: PageIndex

Cherny notes code is highly structured (clear hierarchy, strict syntax, rich naming semantics), which suits agentic search. Unstructured long documents (contracts, reports, prospectuses) are harder. PageIndex addresses this with a hierarchical, tree-style in-context index placed directly in the model's context window: the model navigates the tree, reasons about relevant sections, then reads only those sections—no vector database involved. A financial QA system built this way reportedly achieved industry-best results on high-complexity financial documents.

Takeaway

The shift from RAG to Agentic Search marks a move from memory-centric to reasoning-centric AI: intelligence lies not in what a model stores, but in how well it can explore, verify, and integrate information using a rich toolset. The costs are latency and tokens; the rewards are accuracy, security, auditability, and naturalness.