Open-Retrieval Conversational Question Answering (ORConvQA) — SIGIR 2020
Overview
This entry discusses the SIGIR 2020 paper "Open-Retrieval Conversational Question Answering", published in the ACM Digital Library:
- Paper link: https://dl.acm.org/doi/abs/10.1145/3397271.3401110
- Venue: SIGIR 2020
- Topic areas: Conversational Search, Evaluation of Search Engines
- Datasets: large QA collections such as Natural Questions and MS MARCO, plus conversational QA benchmarks.
- Metrics: MRR, Recall@k, nDCG@10, and answer exact match / F1.
- Baselines: BM25, dense retrieval, cross-encoder reranking, and no-retrieval LLM baselines.
- Ablations: contribution of retrieval depth, reranking, and history modeling to final quality.
- A Survey of Conversational Search (Sep 2025, ACM)
- Engineering Conversational Search Systems: A Review of Applications (arXiv:2407.00997)
- Agentic Conversational Search with Contextualized Reasoning
- ChatRetriever: Adapting LLMs for Generalized and Robust Conversational Dense Retrieval (arXiv:2404.13556)
- CoSearchAgent: A Lightweight Collaborative Search Agent (arXiv:2402.06360)
- Original paper: Open-Retrieval Conversational Question Answering, SIGIR 2020. DOI: 10.1145/3397271.3401110
Problem and Motivation
The paper addresses open-retrieval conversational question answering: answering multi-turn user questions by searching over a large, open collection of documents rather than a small fixed set of passages. This setting combines the challenges of conversational query understanding (resolving coreference and context across turns) with large-scale open retrieval.
Traditional pipelines treat retrieval, ranking, and generation as isolated stages; this line of work motivated more integrated, end-to-end approaches that now underpin modern retrieval-augmented generation (RAG) systems.
Typical System Architecture
The standard technical route described in this research area follows four stages:
1. Input and representation — encode the (rewritten) query, documents, and conversational history into dense or sparse representations. 2. Core modules — a retriever, a reranker, and a reader/answer extractor chained together; optionally memory or history-modeling components. 3. Learning strategy — supervised fine-tuning, contrastive learning, distillation, or reinforcement learning. 4. Inference — single-pass or iterative retrieval, early stopping, and latency/budget control.
Evaluation
Typical experimental setups in this area include:
Exact numerical results should be verified against the original PDF; this post summarizes the experimental design logic based on the abstract and public metadata.
Key Takeaways for Search / Rec / Personalization
1. Architecture: cascaded retrieve → rerank → generate remains mainstream, with agentic paradigms making retrieval strategy itself learnable. 2. Data: high-quality instruction data and session logs are as important as model architecture. 3. Evaluation: offline metrics increasingly diverge from online satisfaction; LLM-as-judge needs human cross-validation. 4. Deployment: latency, cost, interpretability, and safety are hard constraints in production systems.
Engineering Checklist
| Area | Question | Suggestion | |------|----------|------------| | Data | PII in training/index? Version control? | Partitioned indexes, anonymization, rollback-able embedding versions | | Latency | p99 budget? Retrieval steps? | Cascade + early stop, query caching, async reranking | | Quality | Do offline gains translate online? | Interleaving experiments, human audits, citation checks | | Safety | Poisoning/bias from open retrieval? | Source allowlists, adversarial detection, output filtering | | Cost | Token/GPU cost per query? | Small-model routing, distillation, hybrid sparse+dense retrieval |
Glossary
| Term | Meaning | |------|---------| | IR | Information Retrieval | | RAG | Retrieval-Augmented Generation | | nDCG | Normalized Discounted Cumulative Gain | | Agentic Search | Search modeled as sequential decision-making with tool calls | | Gen-IR | Generative Information Retrieval |