Overview
On July 5, 2026, LlamaIndex open-sourced legal-kb, a complete reference application showcasing an agentic retrieval pattern called the Retrieval Harness. The core idea: give an AI agent four file-system-style tools so it can work like a human—find documents, inspect content, verify details—rather than relying on a single vector search returning Top-K chunks.
The Four Tools
| Tool | Capability | Underlying API |
|---|---|---|
| retrieve | Hybrid semantic retrieval + optional rerank + citations | beta.retrieval.retrieve |
| findFiles | Search by filename / substring | beta.retrieval.find |
| readFile | Raw content reads with offset | beta.retrieval.read |
| grepFile | Regex matching with character positions | beta.retrieval.grep |
A system prompt enforces a calling order: the agent must first call findFiles to list files, then retrieve to locate content, and finally readFile/grepFile to confirm exact wording.
The stack: TanStack Start web app + Vercel AI SDK 6 ToolLoopAgent + Prisma (persistence) + WorkOS (auth).
Key Features
- Knowledge base versioning: Re-uploading
nda.pdfcreates coexisting v1, v2, v3 versions, filtered at retrieval time by aversionmetadata field. The KB effectively has git-like semantics—every citation is traceable to a version. - Visual citations: Each retrieved chunk gets a short id (e.g.,
cite:c7f2qa). The agent cites inline, the UI renders clickable citation chips, and clicking opens a screenshot of the source page with a bounding box around the cited text. - Top-K is a black box—users don't know what was retrieved
- Chunk boundaries break semantics—contract clauses, functions, policy sections get split
- No cross-document reasoning—"compare NDA and MSA breach clauses" fails in one retrieval
- No version control—stale citations linger after policy updates
- "What changed in this MSA's breach clauses vs. the 2025 version?" — retrieve + version filter
- "Compare confidentiality terms across these three NDAs" — findFiles + retrieve + grep
- "Find all clauses mentioning force majeure" — grepFile full scan
- "Where exactly is this cited passage?" — visual citation with bounding box
- "RAG is dead, agentic retrieval lives" may be the key paradigm signal for H2 2026. LlamaIndex, the de facto RAG standard-setter, upgraded RAG from single-shot retrieval to multi-step loops—an architectural engine swap, not a patch.
- File-system-style agent tooling is a broader trend:
retrieve/findFiles/readFile/grepFileare AI rewrites offind,cat,grep, andsearch. Cursor, Claude Code, and Vercel AI SDK are converging here—familiar operation semantics beat novel abstractions. - Visual citations with bounding boxes advance answer explainability, helping counter circular-verification bias by making every answer traceable to a specific passage.
- KB versioning applies to any industry with data lifecycles: contracts, financial reports, clinical guidelines, regulations.
- Reference app, not production-ready: legal deployment needs compliance auditing, permission isolation, cross-project retrieval, and agent behavior logs.
- Depends on LlamaCloud (LlamaIndex's commercial service); the
@llamaindex/llama-cloudSDK is closed-source, though the underlying layer can be self-hosted. - Multi-step loops raise costs: one contract query may trigger 4–6 LLM calls.
- Agent autonomy vs. prompt-enforced constraints: the prompt forces findFiles first, but production agents often skip it—determinism vs. flexibility, revisited.
- https://www.marktechpost.com/2026/07/05/llamaindex-legal-kb-agentic-retrieval-over-index-v2-with-retrieve-find-read-and-grep-tools
- https://www.tmtpost.com/agent/ai-article/18893
- https://github.com/run-llama/legal-kb
- https://aihot.virxact.com/items/cmr7ita5g00reslxdrprk52b0
Analysis: From Retrieval Technique to Data Infrastructure
Limits of traditional RAG
The dominant RAG pattern over the past three years: chunk documents → embed → single vector search → Top-K chunks → prompt → answer. Its problems:
Three core shifts in the Retrieval Harness
1. The agent is an operator, not a receiver: It actively decides which files to inspect, what queries to run, and when to read full text. From "automatic" to "manual" transmission.
2. Retrieval is a loop, not a single call: The ToolLoopAgent can iterate—"I wanted X, but this looks like Y, so let me grep again." This is the LLM-era equivalent of grep | head | tail | awk pipelines.
3. The knowledge base is alive, not static: Continuous indexing + versioning means updates propagate while old versions stay queryable—essential for legal, financial, and medical knowledge where history matters.
Fit for legal and finance
Legal documents are cross-file, long, version-sensitive, and citation-precise. Example queries the pattern handles:
Why It Matters
Risks and Open Questions
Conclusion
legal-kb is less a product than a paradigm redefinition: RAG restructured into a three-layer architecture of agent + data pipeline + tool loop. For AI coding, the lesson is that context engineering isn't about stuffing more chunks into a prompt—it's about giving the agent tools to find what it needs. If Cursor, Claude Code, or Cline absorb this Retrieval Harness pattern, it could transform AI-assisted development from "chatting with AI that gives me code" to "collaborating with AI that searches freely, cites precisely, and is version-traceable across the full lifecycle."
This is RAG 2.0.
---
Sources: