Asking Clarification Questions to Handle Ambiguity in Open-Domain QA (CAMBIGNQ, arXiv 2305.13808)

Forum topic · 小凯 · 2026-07-05

Summary

This post summarizes the 2023 paper "Asking Clarification Questions to Handle Ambiguity in Open-Domain QA" (arXiv:2305.13808). Ambiguous questions remain a persistent problem in open-domain question answering because users often cannot formulate questions with a single unique answer. Rather than generating disambiguated questions for every possible interpretation, as Min et al. (2020) did, the authors propose asking the user a clarification question whose response identifies the interpretation that best matches their intent. The paper introduces CAMBIGNQ, a dataset of 5,654 ambiguous questions, each paired with relevant passages, possible answers, and a clarification question. Clarification questions were generated with InstructGPT and then manually revised. The authors define a task pipeline with dedicated evaluation metrics and report 61.3 F1 on ambiguity detection and 40.5 F1 on clarification-based QA, establishing strong baselines for future research. The forum post also provides contextual analysis on information retrieval systems, RAG pipelines, agentic search, evaluation practices, and engineering deployment considerations for search and recommendation systems.

Asking Clarification Questions to Handle Ambiguity in Open-Domain QA (CAMBIGNQ)

Metadata

| Field | Value | |-------|-------| | Paper | Asking Clarification Questions to Handle Ambiguity in Open-Domain QA | | Authors | Dongryeol Lee, Segwang Kim, Minwoo Lee, Hwanhee Lee, Joonsuk Park, Sang-Woo Lee, et al. (7 authors) | | Published | 2023-05-23 | | Source | https://arxiv.org/abs/2305.13808 | | Type | Academic paper | | Category | Search Assistance |

One-line Summary

This work proposes handling ambiguous open-domain questions by asking the user a clarification question, backed by the CAMBIGNQ dataset of 5,654 ambiguous questions and strong baseline results.

Background and Motivation

Ambiguous questions persist in open-domain question answering because formulating a question with a unique answer is often difficult. Min et al. (2020) addressed this by generating disambiguated questions for all possible interpretations — effective, but not ideal for directly answering the user. Instead, this paper proposes asking a clarification question, where the user's response helps identify the interpretation that best aligns with the user's intention.

Key Contributions

CAMBIGNQ dataset: 5,654 ambiguous questions, each with relevant passages, possible answers, and a clarification question. Clarification questions were efficiently generated using InstructGPT and manually revised as needed.
Task pipeline and metrics: A defined pipeline of subtasks (ambiguity detection, clarification-based QA) with appropriate evaluation metrics.
Strong baselines: 61.3 F1 on ambiguity detection and 40.5 F1 on clarification-based QA.

Original Abstract

> Ambiguous questions persist in open-domain question answering, because formulating a precise question with a unique answer is often challenging. Previously, Min et al. (2020) have tackled this issue by generating disambiguated questions for all possible interpretations of the ambiguous question. This can be effective, but not ideal for providing an answer to the user. Instead, we propose to ask a clarification question, where the user's response will help identify the interpretation that best aligns with the user's intention. We first present CAMBIGNQ, a dataset consisting of 5,654 ambiguous questions, each with relevant passages, possible answers, and a clarification question. The clarification questions were efficiently created by generating them using InstructGPT and manually revising them as necessary. We then define a pipeline of tasks and design appropriate evaluation metrics. Lastly, we achieve 61.3 F1 on ambiguity detection and 40.5 F1 on clarification-based QA, providing strong baselines for future work.

Contextual Notes (from the post author)

The post situates the paper within the broader information retrieval landscape: the shift from BM25 and dense retrievers toward generative retrieval, RAG, and agentic search, where "how many retrieval steps to take" becomes a learnable decision. It also lists practical engineering concerns — latency budgets, PII handling in indexed data, online vs. offline metric gaps, retrieval poisoning risks, and per-query token costs — that must be considered when translating paper results into production search systems.

Takeaways

1. Asking a clarification question is a more user-aligned alternative to enumerating all disambiguated interpretations. 2. LLM-generated data (InstructGPT) plus light manual revision can scale dataset construction efficiently. 3. Reported baselines (61.3 F1 ambiguity detection, 40.5 F1 clarification-based QA) leave substantial room for improvement on both tasks.

References

Original paper: Asking Clarification Questions to Handle Ambiguity in Open-Domain QA