A Survey of Conversational Search (ACM, Sep 2025)

This post summarizes and contextualizes an ACM survey on conversational search, published September 2025. Source: https://dl.acm.org/doi/full/10.1145/3759453

Key points

• Motivation: Traditional pipelines separate retrieval, ranking, and generation, which struggles to meet LLM-era demands for natural language interaction, multi-hop reasoning, and up-to-date knowledge. Conversational search must redistribute responsibilities among retrieval, ranking, generation, and tool calling.
• Unified framework: The survey organizes scattered work into a comparable framework, decomposing systems into representation learning, retrievers, rerankers, planners, generators, and feedback mechanisms.
• Four technical mainlines:
• *Dense retrieval*: high recall, low latency; suited for first-stage retrieval.
• *Late interaction* (e.g., ColBERT): higher precision but larger indexes.
• *Generative IR*: directly "generates" documents or docids, simplifying cascades.
• *Agentic search*: models search as sequential decision-making, supporting multi-hop queries and self-reflection.
• LLM integration: RAG, search agents, and tool/API use are flexible and interpretable but introduce latency and error propagation.

Taxonomy

| Dimension | Subtypes | Representative ideas | Strengths | Limitations | |---|---|---|---|---| | Modeling paradigm | Discriminative / generative retrieval | Bi-encoders, cross-encoders, DSI | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization objectives | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce annotations | | Evaluation | Offline / online / human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Gap with real satisfaction |

Evolution timeline

• 2019–2021: BERT reranking and DPR establish neural retrieval.
• 2022–2023: RAG and FreshLLM drive retrieval–generation fusion.
• 2024 onward: Conversational/agentic search and Gen-RecSys explode.
• 2025–2026: RL-trained search agents, Deep Research, and GraphRAG become new growth fronts.

Evaluation practices covered

• Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, recommendation benchmarks.
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, token cost.
• Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.

Evaluation is shifting from static nDCG toward process metrics: task success rate, citation accuracy, and multi-hop reasoning chain completeness.

Insights for search/rec/personalization

1. Architecture: Cascade retrieval + rerank + generation remains mainstream, but the agentic paradigm makes "how many retrievals and what strategy" itself learnable. 2. Data: High-quality instruction data and click/session logs are both critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: The offline–online gap is widening; LLM-as-judge needs cross-validation with human evaluation. 4. Product: Latency, cost, interpretability, and safety are hard industrial constraints—academic benchmarks alone are insufficient.

Open problems

• Lack of unified benchmarks; irreproducible private data.
• LLM evaluation bias; hallucination and safety risks.
• Latency/cost constraints for agentic systems on the open web.
• Cross-lingual (English-centric data) and multimodal generalization remain unverified.
• Future directions: finer-grained process supervision, retrieval–reasoning co-training, enterprise metadata governance, deeper fusion with knowledge graphs, causal and fairness constraints for recommendation.

Engineering checklist (from the post)

| Item | Question | Recommendation | |---|---|---| | Data | PII in training/index? Versioning? | Partitioned indexes, anonymization, rollback-able embedding versions | | Latency | p99 budget? Retrieval steps? | Cascades + early stop, query caching, async reranking | | Quality | Do offline gains translate to online CTR? | Interleaving experiments, human audits, citation verification | | Safety | Poisoning/bias from open retrieval? | Source whitelists, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Small-model routing, distillation, hybrid sparse+dense |

Related entries cross-referenced in the post

• Engineering Conversational Search Systems: A Review of Applications (arXiv 2407.00997)
• Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning
• CTR-Guided Generative Query Suggestion in Conversational Search (EMNLP)
• ChatRetriever: Adapting LLMs for Generalized and Robust Retrieval (arXiv 2404.13556)
• CoSearchAgent: A Lightweight Collaborative Search Agent (arXiv 2402.06360)
• ConvGQR: Generative Query Reformulation for Conversational Search (arXiv 2305.15645)

> Note: Quantitative results should be verified against the original PDF; this post is based on the abstract and public metadata.