Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? (arXiv, June 2024)

Forum topic · 小凯 · 2026-07-05

Summary

This arXiv paper (2406.13121, June 2024) by Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Devendra Singh Sachan, Michael Boratko and colleagues from Google DeepMind investigates whether extending the context window of large language models can replace traditional retrieval-based systems such as RAG pipelines and text-to-SQL. Instead of retrieving a small set of passages, long-context LLMs could in principle read entire corpora directly, eliminating retrieval components and their associated errors. The work examines this question across information retrieval, question answering, and structured data tasks, analyzing where long-context approaches match or fall short of retrieval-augmented architectures in accuracy, efficiency, and cost. The post situates the paper within the broader debate on reallocating responsibilities between retrieval, ranking, generation, and tool use in LLM-era systems, and notes open issues such as evaluation reliability, latency, hallucination, and multilingual or multimodal generalization. Readers should consult the original PDF for quantitative results.

Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?

arXiv: 2406.13121 — June 2024

Overview

This paper, authored by Jinhyuk Lee, Anthony Chen, Zhuyun Dai, Dheeru Dua, Devendra Singh Sachan, Michael Boratko, and others, addresses a fundamental architectural question for LLM-era systems: as context windows grow very large, can long-context language models simply read entire corpora or databases directly, thereby subsuming traditional retrieval components such as RAG pipelines, dense retrieval, and text-to-SQL?

Background and Motivation

Traditional search and RAG pipelines separate retrieval, ranking, and generation into stages. Retrieval narrows a large corpus to a handful of passages before generation, which improves efficiency but introduces cascade errors and limits multi-hop reasoning. With ever-larger context windows, an alternative emerges: feed the whole corpus (or a large slice of it) into the model and let it reason over everything at once — potentially eliminating the retriever entirely and simplifying tasks like text-to-SQL, where the model could scan tables directly instead of writing queries.

Core Questions

Can long-context LLMs match or exceed retrieval-augmented (RAG) approaches on open-domain QA and IR benchmarks?
Can they replace SQL-style structured querying by directly consuming tabular data?
What are the trade-offs in accuracy, latency, and token cost when reading whole corpora instead of retrieved snippets?

Key Takeaways for Practitioners

1. Architecture: Cascaded retrieval + re-ranking + generation remains mainstream, but agentic paradigms increasingly treat "when and how much to retrieve" as a learnable decision. 2. Data: High-quality instruction data and interaction logs remain critical; synthetic data requires care against leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation. 4. Production: Latency, cost, interpretability, and safety constraints dominate industrial deployment — optimizing academic benchmarks alone is insufficient.

Limitations and Open Problems

The broader discussion around this line of work highlights open issues including benchmark fidelity versus real user distributions, English-centric data and unknown cross-lingual generalization, safety risks of agentic systems on the open web, and the engineering realities of context length (attention cost, index update frequency, embedding version compatibility).

Related Entries

A Survey of Graph Retrieval-Augmented Generation for Customized LLMs (arXiv 2501.13958)
A Survey on Retrieval-Augmented Text Generation for LLMs (arXiv 2404.10981)
Agentic RAG: A Survey (arXiv 2501.09136)
RAFT: Adapting Language Models to Domain-Specific RAG
RAG vs. GraphRAG: A Systematic Evaluation (arXiv 2502.11371)

Glossary

| Term | Meaning | |------|---------| | RAG | Retrieval-Augmented Generation | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Modeling search as sequential decisions and tool calls | | Gen-IR | Generative Information Retrieval |

> Note: Quantitative results should be verified against the original paper; this post is based on the abstract and public metadata.