Open-Retrieval Conversational Question Answering, SIGIR 2020
Overview
This entry indexes the SIGIR 2020 paper "Open-Retrieval Conversational Question Answering" (commonly known as ORConvQA).
| Field | Content | |---|---| | Title | Open-Retrieval Conversational Question Answering, SIGIR 2020 | | Source | https://dl.acm.org/doi/abs/10.1145/3397271.3401110 | | Type | Academic paper | | Sections | Conversational Search; Evaluation of Search Engines |
Background and Motivation
The paper targets open-retrieval conversational question answering, where a system answers multi-turn questions by retrieving evidence from a large open corpus rather than a fixed, pre-filtered passage set. Traditional pipelines that separate retrieval, ranking, and generation struggle with user intent understanding across turns, multi-hop reasoning, and incorporation of real-time knowledge — challenges this work addresses at the intersection of conversational search and open-domain information access.
Core Contributions (per the entry's framing)
- A unified perspective on the problem domain, placing scattered related work into a comparable framework.
- A clear decomposition of method components: representation learning, retrievers, rerankers, planners, generators, and feedback mechanisms.
- Reproducible benchmarks, datasets, and taxonomies that lower the entry cost for follow-up research.
- Discussion of interfaces with emerging paradigms such as LLM tool calling, reinforcement learning, and multi-agent collaboration.
- Open problems: evaluation trustworthiness, latency and cost, hallucination and safety, cross-lingual and multimodal extension.
- Datasets: MS MARCO, BEIR, Natural Questions, domain corpora.
- Metrics: nDCG@10, MRR, Recall@k, Hit@k, task success rate, latency and token cost.
- Baselines: BM25, dense retrieval, cross-encoder reranking, retrieval-free LLMs, commercial search APIs.
- Ablations: contribution of retrieval steps, reranking depth, and training data scale.
- A Survey of Conversational Search, Sep 2025, ACM
- Engineering Conversational Search Systems: A Review of Applications
- Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning
- CTR-Guided Generative Query Suggestion in Conversational Search, EMNLP
- ChatRetriever: Adapting Large Language Models for Generalized and Robust Retrieval
- CoSearchAgent: A Lightweight Collaborative Search Agent with Large Language Models
- Original paper: Open-Retrieval Conversational Question Answering, SIGIR 2020. DOI: 10.1145/3397271.3401110
Typical Method Pipeline
1. Input and representation — encode queries, documents, and conversational context into dense or sparse representations. 2. Core modules — retriever, reranker, planner, memory, and tool interfaces, composed in series or parallel. 3. Learning strategies — supervised fine-tuning, contrastive learning, distillation, reinforcement learning, synthetic data bootstrapping. 4. Inference — single-pass or iterative retrieval, parallel sub-queries, early stopping and budget control.
Evaluation
Common setup in this line of work:
Exact quantitative results should be verified against the original PDF.
Takeaways
1. Architecture: cascaded retrieval + reranking + generation remains mainstream; agentic paradigms increasingly treat retrieval policy itself as learnable. 2. Data: high-quality instruction data and click/session logs matter; synthetic data requires leakage and distribution-shift safeguards. 3. Evaluation: the gap between offline metrics and online satisfaction is widening; LLM-as-judge needs cross-validation with human assessment. 4. Productization: latency, cost, explainability, and safety are hard constraints for deployment.