A Survey of Conversational Search (ACM, September 2025)
Overview
This post discusses an ACM survey on conversational search published in September 2025. The original paper is available at: https://dl.acm.org/doi/full/10.1145/3759453
The survey addresses a core question of the LLM era: how to redistribute the responsibilities of retrieval, ranking, generation, and tool calling in modern search systems, where users demand natural-language interaction, multi-hop reasoning, and up-to-date knowledge.
Key points
- Unified framework: The survey organizes scattered work into a comparable taxonomy spanning modeling paradigms, LLM integration, optimization objectives, and evaluation.
- Component breakdown: Clear decomposition of representation learning, retrievers, rerankers, planners, generators, and feedback mechanisms, easing engineering adoption.
- Interfaces with new paradigms: Connections to LLM tool calling, reinforcement learning, and multi-agent collaboration, with paths from research prototypes to industrial systems.
- Open problems: Evaluation trustworthiness, latency/cost, hallucination and safety, and cross-lingual/multimodal scaling.
- Dense retrieval: high recall, low latency; suited for first-stage retrieval.
- Late interaction (e.g., ColBERT): higher precision but larger indexes.
- Generative IR: generates documents directly via tokens or docids, simplifying cascades.
- Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection.
- 2019–2021: BERT reranking and DPR establish neural retrieval foundations.
- 2022–2023: RAG and FreshLLM drive retrieval-generation fusion.
- 2024 onward: Conversational/agentic search and Gen-RecSys explode.
- 2025–2026: RL-trained search agents, Deep Research, and GraphRAG become new growth frontiers.
- Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, public recommendation sets.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, token cost.
- Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
- Note: evaluation is shifting from static nDCG toward process metrics such as task success rate, citation accuracy, and multi-hop reasoning-chain completeness.
- Engineering Conversational Search Systems: A Review of Applications
- Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning
- CTR-Guided Generative Query Suggestion in Conversational Search (EMNLP)
- ChatRetriever: Adapting LLMs for Generalized and Robust Retrieval
- CoSearchAgent: A Lightweight Collaborative Search Agent
- ConvGQR: Generative Query Reformulation for Conversational Search
- Original: A Survey of Conversational Search, ACM, September 2025. https://dl.acm.org/doi/full/10.1145/3759453
Taxonomy
| Dimension | Subcategories | Representative ideas | Strengths | Limitations | |-----------|---------------|---------------------|-----------|-------------| | Modeling paradigm | Discriminative / generative retrieval | Bi-encoders, cross-encoders, DSI, GPT indexing | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization goals | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Label scarcity | | Evaluation | Offline / online / human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Diverges from real satisfaction |
Four main technical lines
Timeline of the field
Benchmarks and evaluation
Insights for Search / Rec / Personalization
1. Architecture: cascade retrieve–rerank–generate remains mainstream, but agentic paradigms make retrieval count and strategy themselves learnable. 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human assessment. 4. Product: latency, cost, explainability, and safety are hard industrial constraints—do not optimize academic benchmarks alone.
Open problems and future directions
The survey highlights insufficient unified benchmarks, non-reproducible private data, LLM evaluation bias, and safety/cost constraints of agentic systems. Future directions include finer-grained process supervision, joint retrieval-reasoning training, enterprise metadata governance, and multimodal/cross-lingual consistency.
Engineering checklist
| Item | Question | Recommendation | |------|----------|----------------| | Data | Does training/index data contain PII? How is versioning managed? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | What is the p99 budget? How many retrieval steps? | Cascade with early stopping, cache hot queries, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelisting, adversarial detection, output filtering | | Cost | Per-query token and GPU usage? | Small-model routing, distillation, hybrid sparse+dense |