Open-Retrieval Conversational Question Answering (SIGIR 2020)

This entry catalogs the SIGIR 2020 paper "Open-Retrieval Conversational Question Answering", filed under the sections *Conversational Search* and *Evaluation of Search Engines*.

• Source: https://dl.acm.org/doi/abs/10.1145/3397271.3401110
• Type: Academic paper

Context and Motivation

The post frames the paper within the challenges facing large-scale search, recommendation, and personalization systems: efficiency, scalability, and user intent understanding in conversational settings. Traditional pipeline approaches that treat retrieval, ranking, and generation as isolated stages struggle to meet modern demands for natural language interaction, multi-hop reasoning, and real-time knowledge access.

The core problem scenarios include:

• Open-domain information access
• Enterprise knowledge retrieval
• Conversational search
• Semantic understanding in recommendation systems
• End-to-end architectures that couple external knowledge sources with generative models

System Architecture Pattern

The post describes a general four-step methodology commonly shared by works in this space:

1. Input and representation: encode queries, documents, and user context as dense or sparse representations, or structured prompts 2. Core modules: retrievers, re-rankers, planners, memory modules, and tool interfaces, chained or combined per task 3. Learning strategies: supervised fine-tuning, contrastive learning, distillation, reinforcement learning (including process rewards), and bootstrapped data synthesis 4. Inference strategies: single-pass retrieval, iterative retrieval, parallel sub-queries, early stopping, and budget control

Evaluation Setup

Typical experimental protocols discussed in this research area:

• Datasets: MS MARCO, BEIR, Natural Questions, domain-specific corpora, public recommendation datasets
• Metrics: nDCG@10, MRR, Recall@k, Hit@k, human preference, task success rate, latency, and token cost
• Baselines: BM25, dense retrieval, cross-encoder re-ranking, retrieval-free LLMs, commercial search APIs
• Ablations: contribution of retrieval steps, re-ranking depth, and training data scale

The post notes that exact quantitative results should be verified against the original PDF, as the summary is based on abstracts and public metadata.

Key Insights for Search / Rec / Personalization

1. Architecture: cascaded retrieval + re-ranking + generation remains mainstream, but agentic paradigms are making "when and how often to retrieve" itself a learnable decision 2. Data: high-quality instruction data and click/session logs are both critical; synthetic data must guard against knowledge leakage and distribution shift 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge should be cross-validated with human evaluation 4. Product: latency, cost, explainability, and safety are hard constraints for industrial deployment — academic benchmarks alone are insufficient

Engineering Checklist

| Concern | Question | Recommendation | |---------|----------|----------------| | Data | PII in training/index? Version control? | Partitioned indexes, anonymization, rollback-capable embedding versions | | Latency | p99 budget? Retrieval steps? | Cascades + early stop, hot-query caching, async re-ranking | | Quality | Do offline gains translate to online CTR/satisfaction? | Interleaving experiments, human audits, citation verification | | Safety | Poisoning/bias via open retrieval? | Source whitelists, adversarial detection, output filtering | | Cost | Token and GPU cost per query? | Route to smaller models, distillation, hybrid sparse+dense |

Related Entries

• A Survey of Conversational Search (Sep 2025, ACM)
• Engineering Conversational Search Systems: A Review of Applications (arXiv 2407.00997)
• Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning
• CTR-Guided Generative Query Suggestion in Conversational Search (EMNLP)
• ChatRetriever: Adapting LLMs for Generalized and Robust Conversational Retrieval (arXiv 2404.13556)
• CoSearchAgent: A Lightweight Collaborative Search Agent (arXiv 2402.06360)

Readers are encouraged to follow the "retrieval → ranking → generation/agentic → evaluation" chain to locate complementary literature, and to cross-validate the paper's contributions against open-source implementations and industrial practice.