From Matching to Generation: A Survey on Generative Information Retrieval (ACM TOIS, Feb 2025)
This forum post catalogs the journal version of the survey "From Matching to Generation: A Survey on Generative Information Retrieval", published in ACM Transactions on Information Systems (TOIS), February 2025.
- Source: https://dl.acm.org/doi/abs/10.1145/3722552
- Type: Survey
- Category: Search Surveys
- The survey systematizes the transition from discriminative *matching*-based retrieval (BM25, bi-encoders, cross-encoders, late interaction such as ColBERT) to *generative* information retrieval, where documents or answers are produced directly by generative models (e.g., DSI-style docid generation, RAG, LLM-based search agents).
- It offers a unifying taxonomy across four dimensions: modeling paradigm, LLM integration (RAG / agents / tool use), optimization objectives (relevance, diversity, freshness), and evaluation (offline, online, human).
- It traces the field's evolution: 2019–2021 BERT re-ranking and DPR established neural retrieval; 2022–2023 RAG and FreshLLM fused retrieval with generation; 2024 onward saw the rise of conversational/agentic search and generative recommendation; 2025+ directions include RL-trained search agents, Deep Research, and GraphRAG.
- Dense retrieval: high recall, low latency; suited for first-stage retrieval, but sensitive to domain shift and long-tail queries.
- Late interaction (e.g., ColBERT): higher accuracy but larger indexes; document representations are not cheaply pre-computable like bi-encoders.
- Generative IR: directly "generates" documents or identifiers, simplifying cascades but complicating index updates.
- Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection.
- Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, recommendation benchmarks.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, token cost.
- Baselines: BM25, dense retrieval, cross-encoder re-ranking, retrieval-free LLMs, commercial search APIs.
- Evaluation is shifting from static ranking metrics toward process-oriented measures such as task success rate, citation accuracy, and multi-hop reasoning chain completeness.
- A Comprehensive Survey on Reinforcement Learning-based Agentic Search (arXiv:2510.16724)
- A Survey of Conversational Search (arXiv:2410.15576)
- A Survey of LLM Empowered Agents for Recommendation (arXiv:2503.05659)
- A Survey of Model Architectures in Information Retrieval (arXiv:2502.14822)
- A Survey on Knowledge-Oriented Retrieval-Augmented Generation (arXiv:2503.10677)
Key points
Taxonomy highlights
| Dimension | Subtypes | Representative ideas | Strengths | Limitations | |---|---|---|---|---| | Modeling paradigm | Discriminative vs. generative retrieval | Bi-encoders, cross-encoders, DSI | Mature, scalable | Semantic drift, index update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce annotations | | Evaluation | Offline / online / human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Diverges from real satisfaction |
Four main research lines
Benchmarks and evaluation trends
Insights and open problems
1. Architecture: cascade retrieve–rerank–generate remains mainstream, but agentic paradigms make "when and how often to retrieve" itself learnable. 2. Data: high-quality instruction data and click/session logs remain critical; synthetic data risks leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human assessment. 4. Deployment: latency, cost, explainability, and safety are hard constraints for industrial systems.
Open problems include lack of unified benchmarks, non-reproducible private data, LLM evaluation bias, and the safety/cost constraints of agentic systems. Future directions: fine-grained process supervision, joint retrieval–reasoning training, multimodal and cross-lingual consistency, deeper fusion with knowledge graphs and structured databases.
Engineering checklist (from the post)
| Item | Question | Suggestion | |---|---|---| | Data | PII in training/index? Versioning? | Partitioned indexes, sanitization, rollback-safe embeddings | | Latency | p99 budget? Retrieval steps? | Cascades with early stopping, query caching, async re-ranking | | Quality | Do offline gains transfer online? | Interleaving experiments, human audits, citation checks | | Safety | Poisoning/bias via open retrieval? | Source whitelisting, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Small-model routing, distillation, hybrid sparse-dense retrieval |