Dense Text Retrieval Based on Pretrained Language Models: A Survey (ACM, Feb 2024)
Overview
| Field | Content | |------|------| | Title | Dense Text Retrieval Based on Pretrained Language Models: A Survey | | Published | Feb 2024, ACM | | Source | https://dl.acm.org/doi/abs/10.1145/3637870 | | Type | Survey |
This survey systematically reviews dense text retrieval methods based on pretrained language models, covering background, taxonomy, representative work, benchmarks, and open problems in the era of large language models (LLMs).
Key points
- Provides a unified perspective that organizes scattered dense-retrieval research into a comparable framework.
- Decomposes method components: representation learning, retrievers, re-rankers, planners, generators, and feedback mechanisms.
- Offers reproducible benchmarks, datasets, and taxonomy tables, lowering the entry barrier for new researchers.
- Discusses interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, plus paths from research prototypes to industrial systems.
- Lists open problems: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual / multimodal extension.
- Dense retrieval: high recall, low latency; suited for first-stage retrieval.
- Late interaction (e.g., ColBERT): higher precision but larger indexes.
- Generative IR: directly "generates" documents via tokens or docids, simplifying the cascade.
- Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection.
- 2019-2021: BERT re-ranking and DPR establish neural retrieval foundations.
- 2022-2023: RAG and FreshLLM drive retrieval-generation fusion.
- 2024+: conversational / agentic search and generative recommendation (Gen-RecSys) surge.
- 2025-2026 (outlook): RL-trained search agents, Deep Research, and GraphRAG become new growth areas.
- Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, recommendation benchmarks.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, token cost.
- Baselines: BM25, dense retrieval, cross-encoder re-ranking, retrieval-free LLMs, commercial search APIs.
- Ablations: contribution of retrieval steps, re-ranking depth, training data scale.
- A Comprehensive Survey on Reinforcement Learning-based Agentic Search
- A Survey of Conversational Search, Oct 2024, arXiv
- A Survey of LLM Empowered Agents for Recommendation
- A Survey of Model Architectures in Information Retrieval, Jan 2025
- A Survey on AI Search with Large Language Models, July 2025
- A Survey on Knowledge-Oriented Retrieval-Augmented Generation, Mar 2025
- Original: *Dense Text Retrieval Based on Pretrained Language Models: A Survey*, ACM, Feb 2024. https://dl.acm.org/doi/abs/10.1145/3637870
Taxonomy
| Dimension | Sub-class | Representative ideas | Strengths | Limitations | |------|------|------|------|------| | Modeling paradigm | Discriminative / generative retrieval | Dual-tower, cross-encoder, DSI, GPT-based indexing | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization objective | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce annotations | | Evaluation | Offline / Online / Human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Gap with real satisfaction |
Main research lines
Timeline
Evaluation paradigm
Quantitative results should be verified against the original paper's tables.
Insights for search / recommendation / personalization
1. Architecture: cascade (retrieve + re-rank + generate) remains mainstream, but the agentic paradigm makes retrieval count and strategy learnable. 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data must guard against leakage and distribution shift. 3. Evaluation: the offline/online gap widens; LLM-as-judge needs cross-validation with human assessment. 4. Product: latency, cost, interpretability, and safety are hard constraints for industrial deployment—do not optimize academic benchmarks alone.
Open problems and future directions
Insufficient unified benchmarks, non-reproducible private data, LLM evaluation bias, and safety/cost constraints of agentic systems. Future work includes finer-grained process supervision, joint retrieval-reasoning training, enterprise metadata governance, and multimodal / cross-lingual consistency.
Engineering checklist
| Item | Question | Suggestion | |--------|------|------| | Data | PII in training/index? Version control? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascade + early stop, query caching, async re-ranking | | Quality | Does offline gain translate to online CTR/satisfaction? | Interleaving experiments, human audit, citation checks | | Safety | Poisoning/bias from open retrieval? | Source whitelisting, adversarial detection, output filtering | | Cost | Per-query tokens and GPU usage? | Small-model routing, distillation, hybrid sparse+dense |
Glossary
| Term | Meaning | |------|------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Search modeled as sequential decisions and tool calls | | Gen-IR | Generative Information Retrieval |