Engineering Conversational Search Systems: A Review of Applications, Architectures, and Functional Components

Authors: Phillip Schneider, Wessel Poelman, Michael Rovatsos, Florian Matthes Published: 2024-07-01 Source: https://arxiv.org/abs/2407.00997 Type: Survey

One-line Summary

A systematic literature review of conversational search systems that maps real-world application scenarios, system architectures, and functional components, culminating in a layered architecture framework contextualized by the rapid progress of large language models.

Background and Motivation

Conversational search systems enable information retrieval via natural language interactions, aiming to maximize users' information gain over multiple dialogue turns. The growing prevalence of conversational interfaces challenges traditional information retrieval approaches, which typically treat retrieval, ranking, and generation as separate pipeline stages. This makes it increasingly important to understand the engineering process behind these systems. The authors conducted a systematic literature review to investigate the links between theoretical studies and technical implementations of conversational search systems, covering open-domain information access, enterprise knowledge retrieval, and end-to-end architectures that combine external knowledge sources with generative models.

Core Contributions

• A unified perspective that consolidates dispersed related work into a comparable framework.
• A clear decomposition of method components (representation learning, retrievers, rerankers, planners, generators, feedback mechanisms) to support engineering practice.
• Reproducible baselines, datasets, and classification tables that lower the entry barrier for subsequent research.
• Discussion of interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration, and the path from research prototypes to industrial systems.
• Explicitly listed open problems: evaluation trustworthiness, latency and cost, hallucination and safety, and cross-lingual/multimodal extension.

Taxonomy of Approaches

| Dimension | Subcategories | Representative Ideas | Strengths | Limitations | |-----------|---------------|----------------------|-----------|-------------| | Modeling paradigm | Discriminative / generative retrieval | Bi-encoders, cross-encoders, DSI, GPT indexing | Mature, scalable | Semantic drift, update cost | | LLM integration | RAG / Agent / Tool-use | Retrieval augmentation, search agents, API calls | Flexible, interpretable | Latency, error propagation | | Optimization targets | Relevance / diversity / freshness | Multi-objective LTR, RLHF, online learning | Business-aligned | Annotation scarcity | | Evaluation | Offline / Online / Human | nDCG, MRR, LLM-as-judge, A/B testing | Comparable | Deviates from real satisfaction |

The survey generally juxtaposes four main lines of work:

• Dense retrieval: high recall, low latency; suited for first-stage retrieval.
• Late interaction (e.g., ColBERT): higher accuracy but larger indexes.
• Generative IR: directly "generates" documents via tokens or docids, simplifying cascades.
• Agentic search: models search as sequential decision-making, supporting multi-hop reasoning and self-reflection.

Timeline of Research Evolution

• 2019–2021: BERT reranking and DPR lay the foundation for neural retrieval.
• 2022–2023: RAG and FreshLLMs drive retrieval-generation fusion.
• 2024 onward: Conversational/agentic search and generative recommender systems (Gen-RecSys) surge.
• 2025–2026: RL-trained search agents, Deep Research, and GraphRAG become new growth areas.

Evaluation Paradigms

Typical benchmarks and trends covered include:

• Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora, public recommendation datasets.
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost.
• Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
• Ablations: contributions of individual modules (retrieval steps, reranking depth, training data scale) to final quality.

Specific numerical results should be verified against the original PDF tables.

Key Insights

1. Architecture: Cascaded retrieval + reranking + generation remains mainstream, but the agentic paradigm is making "retrieval frequency and policy" itself a learnable object. 2. Data: High-quality instruction data and click/session logs are equally critical; synthetic data must guard against knowledge leakage and distribution shift. 3. Evaluation: The gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation. 4. Product: Latency, cost, explainability, and safety policies are hard constraints for industrial deployment—academic benchmarks alone are insufficient.

Open Problems and Future Directions

The authors point to insufficient unified benchmarks, non-reproducible private data, LLM evaluation bias, and the safety and cost constraints of agentic systems. Future work includes finer-grained process supervision, joint retrieval-reasoning training, enterprise metadata governance, and multimodal/cross-lingual consistency.

Related Entries

• A Survey of Conversational Search, Sep 2025, ACM
• Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning
• CTR-Guided Generative Query Suggestion in Conversational Search, EMNLP
• ChatRetriever: Adapting Large Language Models for Generalized and Robust Retrieval
• CoSearchAgent: A Lightweight Collaborative Search Agent with Large Language Models
• ConvGQR: Generative Query Reformulation for Conversational Search

Original Abstract

> Conversational search systems enable information retrieval via natural language interactions, with the goal of maximizing users' information gain over multiple dialogue turns. The increasing prevalence of conversational interfaces adopting this search paradigm challenges traditional information retrieval approaches, stressing the importance of better understanding the engineering process of developing these systems. We undertook a systematic literature review to investigate the links between theoretical studies and technical implementations of conversational search systems. Our review identifies real-world application scenarios, system architectures, and functional components. We consolidate our results by presenting a layered architecture framework and explaining the core functions of conversational search systems. Furthermore, we reflect on our findings in light of the rapid progress in large language models, discussing their capabilities, limitations, and directions for future research.

Glossary

| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | LTR | Learning to Rank | | nDCG | Normalized Discounted Cumulative Gain, a ranking quality metric | | Agentic Search | Paradigm modeling search as sequential decision-making and tool calls | | Gen-IR | Generative Information Retrieval |