Engineering Conversational Search Systems: A Review of Applications, Architectures, and Functional Components
Authors: Phillip Schneider, Wessel Poelman, Michael Rovatsos, Florian Matthes Published: 2024-07-01 Source: https://arxiv.org/abs/2407.00997 Type: Survey (Systematic Literature Review)
Overview
Conversational search systems enable information retrieval via natural language interactions, with the goal of maximizing users' information gain over multiple dialogue turns. The increasing prevalence of conversational interfaces adopting this search paradigm challenges traditional information retrieval approaches, stressing the importance of better understanding the engineering process of developing these systems.
The authors undertook a systematic literature review to investigate the links between theoretical studies and technical implementations of conversational search systems. The review identifies real-world application scenarios, system architectures, and functional components, consolidated into a layered architecture framework explaining the core functions of conversational search systems.
Background and Motivation
Traditional pipeline-style approaches treat retrieval, ranking, and generation as separate stages, which struggles to meet the demands of the LLM era: natural language interaction, multi-hop reasoning, and access to real-time knowledge. This survey systematically maps the theoretical and practical boundaries of the conversational search field, covering open-domain information access, enterprise knowledge retrieval, semantic understanding in recommendation, and end-to-end architectures that combine external knowledge sources with generative models.
Key Contributions
- A unified perspective that brings scattered related work into a comparable framework
- A clear decomposition of method components: representation learning, retrievers, rerankers, planners, generators, and feedback mechanisms
- Reproducible benchmarks, datasets, and taxonomies that lower the entry barrier for subsequent research
- Discussion of interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, with migration paths from research prototypes to industrial systems
- Explicitly stated open problems: evaluation reliability, latency and cost, hallucination and safety, and cross-lingual/multimodal extension
- Dense retrieval: high recall, low latency; suited for first-stage retrieval
- Late interaction (e.g., ColBERT): higher accuracy but larger indexes
- Generative IR: directly "generates" documents via tokens or docids, simplifying the cascade
- Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection
- 2019–2021: BERT reranking and DPR establish neural retrieval foundations
- 2022–2023: RAG and FreshLLMs drive retrieval–generation fusion
- 2024 onward: Conversational/agentic search and Gen-RecSys proliferate
- 2025–2026: RL-trained search agents, Deep Research, and GraphRAG become new growth areas
- Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost
- Baselines: BM25, dense retrieval, cross-encoder reranking, no-retrieval LLMs, commercial search APIs
- Ablations: contributions of retrieval steps, rerank depth, and training data scale
- A Survey of Conversational Search (ACM, Sep 2025)
- Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning
- CTR-Guided Generative Query Suggestion in Conversational Search (EMNLP)
- ChatRetriever: Adapting Large Language Models for Generalized and Robust Retrieval
- CoSearchAgent: A Lightweight Collaborative Search Agent with Large Language Models
- ConvGQR: Generative Query Reformulation for Conversational Search
Taxonomy of Approaches
| Dimension | Subcategories | Representative Ideas | Strengths | Limitations | |---|---|---|---|---| | Modeling paradigm | Discriminative / generative retrieval | Bi-encoders, cross-encoders, DSI, GPT indexing | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calling | Flexible, interpretable | Latency, error propagation | | Optimization objective | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce annotations | | Evaluation | Offline / Online / Human | nDCG, MRR, LLM-as-judge, A/B testing | Comparable | Diverges from real satisfaction |
Four main technical lines are typically contrasted:
Timeline of the Field
Evaluation Paradigms
Quantitative results should be verified against the original PDF tables.
Key Takeaways for Search and Recommendation
1. Architecture: Cascaded retrieval + rerank + generation remains mainstream, but the agentic paradigm makes retrieval count and strategy themselves learnable 2. Data: High-quality instruction data and click/session logs are equally critical; synthetic data must guard against knowledge leakage and distribution shift 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human assessment 4. Product: Latency, cost, interpretability, and safety are hard constraints for industrial deployment—academic benchmarks alone are insufficient
Open Problems and Future Directions
Highlighted gaps include insufficient unified benchmarks, non-reproducible private data, LLM evaluation bias, and the safety and cost constraints of agentic systems. Future work includes finer-grained process supervision, joint retrieval–reasoning training, enterprise metadata governance, and multimodal and cross-lingual consistency.
Engineering Checklist
| Item | Question | Recommendation | |---|---|---| | Data | PII in training/index? Version management? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? How many retrieval steps? | Cascade + early stopping, hot-query caching, async reranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelists, adversarial detection, output filtering | | Cost | Per-query tokens and GPU usage? | Route to small models, distillation, hybrid sparse+dense |
Glossary
| Term | Meaning | |---|---| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Search modeled as sequential decision-making and tool calling | | Gen-IR | Generative Information Retrieval |
Original Abstract
> Conversational search systems enable information retrieval via natural language interactions, with the goal of maximizing users' information gain over multiple dialogue turns. The increasing prevalence of conversational interfaces adopting this search paradigm challenges traditional information retrieval approaches, stressing the importance of better understanding the engineering process of developing these systems. We undertook a systematic literature review to investigate the links between theoretical studies and technical implementations of conversational search systems. Our review identifies real-world application scenarios, system architectures, and functional components. We consolidate our results by presenting a layered architecture framework and explaining the core functions of conversational search systems. Furthermore, we reflect on our findings in light of the rapid progress in large language models, discussing their capabilities, limitations, and directions for future research.