Retrieval-Augmented Generation for Large Language Models: A Survey (2023)
This post summarizes a 2023 survey, "Retrieval-Augmented Generation for Large Language Models: A Survey", available via the BAAI simg paper repository:
https://simg.baai.ac.cn/paperfile/25a43194-c74c-4cd3-b60f-0a1f27f8b8af.pdf
Key points
- The survey addresses RAG challenges in large-scale search, recommendation, and personalization systems: efficiency, scalability, and user-intent understanding. Traditional pipelines separate retrieval, ranking, and generation, which struggles to meet LLM-era demands for natural-language interaction, multi-hop reasoning, and up-to-date knowledge.
- It provides a unified perspective that brings scattered related work into a comparable framework, decomposing methods into components such as representation learning, retrievers, rerankers, planners, generators, and feedback mechanisms.
- It discusses interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, and outlines the path from research prototypes to industrial systems.
- Dense retrieval: high recall, low latency; suitable for first-stage retrieval.
- Late interaction (e.g., ColBERT): higher precision but larger indexes.
- Generative IR: directly "generates" documents via tokens or docids, simplifying cascades.
- Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection.
- 2019–2021: BERT reranking and DPR establish neural retrieval foundations.
- 2022–2023: RAG and FreshLLM drive retrieval-generation fusion.
- 2024 onward: conversational/agentic search and Gen-RecSys surge.
- 2025–2026: RL-trained search agents, Deep Research, and GraphRAG become new growth areas.
- Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, public recommendation sets.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost.
- Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
- Insufficient unified benchmarks; non-reproducible private data.
- LLM evaluation bias.
- Safety and cost constraints of agentic systems on the open web.
- Future directions: finer-grained process supervision, joint retrieval-reasoning training, enterprise metadata governance, multimodal and cross-lingual consistency.
- IR: Information Retrieval
- RAG: Retrieval-Augmented Generation
- LTR: Learning to Rank
- nDCG: Normalized Discounted Cumulative Gain
- Agentic Search: search modeled as sequential decision-making and tool invocation
- Gen-IR: Generative Information Retrieval
- A Comprehensive Survey on Reinforcement Learning-based Agentic Search (arXiv:2510.16724)
- A Survey of Conversational Search, Oct 2024 (arXiv:2410.15576)
- A Survey of LLM-Empowered Agents for Recommendation (arXiv:2503.05659)
- A Survey of Model Architectures in Information Retrieval, Jan 2025 (arXiv:2502.14822)
- A Survey on AI Search with Large Language Models, July 2025
- A Survey on Knowledge-Oriented Retrieval-Augmented Generation, Mar 2025 (arXiv:2503.10677)
Taxonomy
| Dimension | Subcategories | Representative ideas | Strengths | Limitations | |---|---|---|---|---| | Modeling paradigm | Discriminative / generative retrieval | Dual-tower, cross-encoders, DSI, GPT indexing | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization objectives | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce annotations | | Evaluation | Offline / online / human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Diverges from real satisfaction |
Four technical lines
Timeline of the field
Evaluation and benchmarks
Typical datasets and metrics covered include:
Note: concrete numeric results should be verified against the original PDF.
Main takeaways
1. Architecture: cascaded retrieval + rerank + generation remains mainstream, but the agentic paradigm makes "when and how many times to retrieve" itself learnable. 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data risks knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Production: latency, cost, explainability, and safety are hard constraints — do not optimize only academic benchmarks.
Open problems
Engineering checklist
| Item | Question | Suggestion | |---|---|---| | Data | PII in training/index? Version management? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? How many retrieval steps? | Cascade + early stopping, caching hot queries, async reranking | | Quality | Do offline gains transfer to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelisting, adversarial detection, output filtering | | Cost | Token and GPU usage per query? | Small-model routing, distillation, hybrid sparse+dense |