Engineering Conversational Search Systems: A Review of Applications, Architectures, and Functional Components
Source: arXiv:2407.00997 | Authors: Phillip Schneider, Wessel Poelman, Michael Rovatsos, Florian Matthes | Published: 2024-07-01 | Type: Survey
Original Abstract
> Conversational search systems enable information retrieval via natural language interactions, with the goal of maximizing users' information gain over multiple dialogue turns. The increasing prevalence of conversational interfaces adopting this search paradigm challenges traditional information retrieval approaches, stressing the importance of better understanding the engineering process of developing these systems. We undertook a systematic literature review to investigate the links between theoretical studies and technical implementations of conversational search systems. Our review identifies real-world application scenarios, system architectures, and functional components. We consolidate our results by presenting a layered architecture framework and explaining the core functions of conversational search systems. Furthermore, we reflect on our findings in light of the rapid progress in large language models, discussing their capabilities, limitations, and directions for future research.
Background and Motivation
In large-scale search, recommendation, and personalization systems, conversational search has long faced challenges around efficiency, scalability, and user intent understanding. Traditional pipeline approaches tend to separate retrieval, ranking, and generation, which struggles to meet LLM-era demands for natural-language interaction, multi-hop reasoning, and real-time knowledge. This survey systematically maps the intersection of theory and engineering practice in conversational search, covering open-domain information access, enterprise knowledge retrieval, semantic understanding in recommendation, and end-to-end architectures that couple external knowledge sources with generative models.
Core Contributions
- A unified perspective that brings scattered related work into a comparable framework.
- A clear decomposition of method components (representation learning, retrievers, re-rankers, planners, generators, feedback mechanisms) to ease engineering adoption.
- Reproducible benchmarks, datasets, and taxonomies that lower the entry cost for new researchers.
- Discussion of interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, including paths from research prototypes to industrial systems.
- Explicit open problems: evaluation trustworthiness, latency and cost, hallucination and safety, cross-lingual and multimodal extension.
- Dense retrieval: high recall, low latency; suited for first-stage retrieval.
- Late interaction (e.g., ColBERT): higher accuracy but larger indexes.
- Generative IR: directly "generates" documents via tokens or docids, simplifying cascades.
- Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection.
- 2019–2021: BERT re-ranking and DPR lay the foundation of neural retrieval.
- 2022–2023: RAG and FreshLLMS drive retrieval-generation fusion.
- 2024 onward: conversational / agentic search and Gen-RecSys take off.
- 2025–2026 (projected): RL-trained search agents, Deep Research, and GraphRAG become new growth areas.
- Datasets: MS MARCO, BEIR, Natural Questions, domain corpora, public recommendation sets.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency and token cost.
- Baselines: BM25, dense retrieval, cross-encoder re-ranking, retrieval-free LLMs, commercial search APIs.
- Ablations: contribution of each module (retrieval steps, re-ranking depth, training data scale) to final quality.
- IR: Information Retrieval
- RAG: Retrieval-Augmented Generation
- LTR: Learning to Rank
- nDCG: Normalized Discounted Cumulative Gain, a ranking quality metric
- Agentic Search: modeling search as sequential decisions and tool calls
- Gen-IR: Generative Information Retrieval
- A Survey of Conversational Search (ACM, Sep 2025)
- Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning
- CTR-Guided Generative Query Suggestion in Conversational Search (EMNLP)
- ChatRetriever: Adapting LLMs for Generalized and Robust Retrieval
- CoSearchAgent: A Lightweight Collaborative Search Agent
- ConvGQR: Generative Query Reformulation for Conversational Search
Taxonomy of Approaches
| Dimension | Sub-categories | Representative ideas | Strengths | Limitations | |---|---|---|---|---| | Modeling paradigm | Discriminative / generative retrieval | Bi-encoders, cross-encoders, DSI, GPT indexing | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization goals | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Scarce annotations | | Evaluation | Offline / Online / Human | nDCG, MRR, LLM-as-judge, A/B tests | Comparable | Gap with real satisfaction |
Main Technical Lines Compared
Timeline of the Field
Evaluation Paradigms
Exact numerical results should be verified against the original PDF tables.
Key Insights
1. Architecture: cascaded retrieval + re-ranking + generation remains mainstream, but the agentic paradigm is making "how many times and how to retrieve" itself a learnable object. 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human assessment. 4. Product: latency, cost, explainability, and safety are hard constraints for industrial deployment — one cannot optimize only academic benchmarks.
Open Problems and Future Directions
The authors and related surveys point to: insufficient unified benchmarks, irreproducibility on private data, LLM evaluation bias, and the safety and cost constraints of agentic systems. Future work includes finer-grained process supervision, joint retrieval-reasoning training, enterprise metadata governance, and multimodal and cross-lingual consistency.
Engineering Checklist
| Item | Question | Recommendation | |---|---|---| | Data | PII in training/index? Version management? | Partitioned indexes, de-identification, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascade with early stopping, hot-query caching, async re-ranking | | Quality | Does offline gain translate to online CTR/satisfaction? | Interleaving experiments, human audit samples, citation verification | | Safety | Does open retrieval introduce poisoning/bias? | Source whitelisting, adversarial detection, output filtering | | Cost | Per-query tokens and GPU usage? | Small-model routing, distillation, hybrid sparse+dense |
Glossary
Recommended Actions
1. Researchers: reproduce the core comparisons; check whether statistical significance and compute costs are reported. 2. Engineers: extract pluggable modules (encoders, re-rankers, planners) and assess integration costs with existing stacks. 3. Product managers: focus on user-perceivable benefits (latency, answer trustworthiness, multi-turn consistency) rather than offline nDCG alone.