DiscoBench: When Search Agents Should Ask — Clarification-Aware Deep Search

Forum topic · 小凯 · 2026-07-06

Summary

DiscoBench is a new benchmark from Tencent Hunyuan and Tsinghua University that exposes a critical blind spot in today's LLM-powered search agents: they lack clarification awareness. When queries become ambiguous during multi-step search, agents typically keep searching rather than asking the user, causing answers to drift off target. Unlike prior benchmarks that treat ambiguity as a static query property, DiscoBench models it as a phenomenon that dynamically emerges and propagates along reasoning chains. Built from real web search scenarios, the dataset spans 11 domains with 211 multi-turn tasks and 463 ambiguity instances across four types (entity, attribute, scope, and relation ambiguity), injected at natural positions via a two-stage, human-verified pipeline. A four-dimensional evaluation framework measures task success rate, ambiguity detection precision (F1), interaction strategy rationality, and cost efficiency, supported by a user simulator that provides progressive clues and partial answers. Experiments on mainstream models (e.g., GPT-4, Claude) show they correctly detect only about 30-40% of ambiguity nodes and that proactively requesting clarification significantly outperforms persistent searching. The findings imply agent frameworks should add a 'clarify' action, ambiguity detection modules, and clarification-first decision policies.

Overview

Paper: When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search (arXiv:2606.27669) Authors: Yiling Tao, Shihan Deng, Meiling Tao, Pengzhi Wei, Zhichao Hu, Zhihao Zhu Institutions: Hunyuan, Tencent; Shenzhen International Graduate School, Tsinghua University

Current LLM search agents (Perplexity, Google Deep Research, open-source frameworks) share a common flaw: when queries become ambiguous, they keep searching instead of stopping to ask. Example: a user asks for the latest "Apple" products — the agent searches for Apple Inc. news, but the user meant the fruit. This is a lack of *clarification awareness*, caused by:

Benchmarks that only score final answers, not ambiguity handling
Frameworks whose action space contains only "search" and "answer"
Training data lacking "ask-first, then search" interaction patterns

Key Insight: Ambiguity Propagates Dynamically

Existing benchmarks either ignore ambiguity, treat it as a static query attribute, or test it only in closed sandboxes. DiscoBench's core contribution: ambiguity is not a static property of query terms — it emerges and amplifies dynamically across multi-step reasoning chains. For example, "best movie of 2025" raises cascading questions: best by box office or ratings? By release year or award year? Conflicting data introduces further ambiguity mid-search.

Dataset Construction

11 real-world domains (film, tech, history, geography, sports, literature, etc.)
211 samples (complete multi-turn search tasks)
463 ambiguity instances (~2.2 per task)
4 ambiguity types: entity ("Apple"), attribute (criteria for "best"), scope (which "2025"?), relation (unclear entity relationships)

Two-stage pipeline: (1) collect high-quality multi-hop reasoning chains manually; (2) inject ambiguity at natural, plausible positions in the chain, with each injection verified by at least 2 annotators and paired with progressive user clues that distinguish target entities.

Four-Dimensional Evaluation

1. Task Success Rate — final answer correctness, in richer context 2. Ambiguity Detection Precision — true/false positives/negatives, scored by F1 3. Interaction Strategy Rationality — whether choosing search / clarify / answer at each node is justified 4. Cost Efficiency — interaction turns, API calls, and the balance of asking too much vs. too little

A user simulator provides progressive clues consistent with real user behavior, including incomplete answers, allowing agents to ask follow-ups.

Experimental Findings

Clarification-aware agents achieve significantly higher task success rates than agents that search straight through
Even strong models (GPT-4, Claude series) mostly keep searching rather than asking
Mainstream models correctly detect only ~30-40% of the 463 ambiguity nodes, with many false negatives (missing ambiguity) and some false positives (unnecessary questions)
The most dangerous failure mode: searching instead of clarifying, compounding wrong assumptions
Agents that clarify spend fewer total turns overall despite asking questions

Why models fail to ask: training data bias toward unambiguous queries, missing evaluation incentives, no "clarify" action in frameworks, and LLM overconfidence.

Why It Matters

1. Conversational retrieval: moving from one-shot query→result to interactive loops resembling human research workflows 2. Deployment-critical: enterprises may see "90% accuracy" on existing benchmarks while users complain about off-target answers — the failures are invisible to current evaluation 3. Reusable methodology: the four metrics capture agent self-awareness, decision quality, and cost balance better than final-answer accuracy alone 4. Framework design: add a "clarify" action, ambiguity-detection modules, clarification-first policies, and simulators for training interaction strategies

Limitations & Future Work

Dataset is primarily Chinese; cross-lingual benchmarks needed
Only 4 ambiguity types; metaphor, cultural, and domain jargon ambiguity remain
Simulator clues are pre-scripted; handling confused or wrong user answers is harder
Ambiguity propagation across multi-agent collaboration is an open problem
Future: fine-tuning on DiscoBench feedback, adding ambiguity recognition as an RLHF reward signal

Bottom Line

A search agent's biggest failure is not failing to find answers — it's not knowing what it's asking. DiscoBench models ambiguity as a dynamic, propagating phenomenon and shows that one clarifying question often beats ten more searches: a key step from one-way retrieval tools toward conversational research assistants.

Reference: arXiv:2606.27669 · Code: to be released