DiscoBench: When Search Agents Should Ask — Benchmarking Clarification-Aware Deep Search

Overview

Paper: When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search Authors: Yiling Tao, Shihan Deng, Meiling Tao, Pengzhi Wei, Zhichao Hu, Zhihao Zhu Institutions: Hunyuan (Tencent); Shenzhen International Graduate School, Tsinghua University Paper: https://arxiv.org/abs/2606.27669 Code: To be released

One-line takeaway: current search agents lack a "sense of direction" — when facing ambiguity they keep searching instead of asking the user, and drift further off-course as a result. DiscoBench is the first benchmark to model ambiguity as a dynamically propagating phenomenon and demonstrates that proactively asking one clarifying question beats blindly searching to the end.

The Blind Spot: Agents Don't Ask for Directions

Modern search agents (Perplexity, Google Deep Research, open-source frameworks) share a common flaw: they never stop to confirm when ambiguity arises — they just keep searching.

Example: a user asks for "Apple's latest product." The agent searches and returns iPhone/Mac news, but the user actually meant the fruit. If the agent asked "Do you mean Apple Inc. or the fruit?" the result would be far more accurate.

This gap in *clarification awareness* persists because:

• Existing benchmarks only score final answers, not ambiguity awareness
• Agent action spaces typically contain only "search" and "answer" — no "clarify"
• Training data rarely includes "clarify first, then search" interaction patterns

Core Innovation: Ambiguity as Propagation, Not Property

Existing benchmarks have three limitations: 1. Ignoring ambiguity — mostly single, well-defined queries 2. Static ambiguity — treating ambiguity as a static property of query words, when in reality it dynamically propagates along multi-step reasoning chains 3. Closed sandboxes — not grounded in real web search

DiscoBench's insight: ambiguity emerges and amplifies during search. E.g., "best movie of 2025" — best by box office, ratings, or awards? 2025 as release year or award year? Conflicting data between sources creates new ambiguity mid-search. Agents must identify ambiguity at each reasoning node and decide whether to search, ask, or answer.

Dataset Construction

• 11 real-world domains: movies, technology, history, geography, sports, literature, etc.
• 211 samples: complete multi-turn search tasks
• 463 ambiguity instances: ~2.2 ambiguity nodes per task
• 4 ambiguity types: entity ("Apple"), attribute (criteria for "best"), scope (which "2025"), relation (unclear entity relations)

Two-Stage Pipeline

1. Seed collection: manually curated high-quality multi-hop reasoning chains across domains 2. Ambiguity injection: ambiguities injected only at naturally plausible positions in the chain, each verified by at least 2 annotators, with progressive user cues generated that distinguish target entities

Evaluation Framework: Four Dimensions

1. Task Success Rate — final answer correctness, placed in richer context 2. Ambiguity Detection Precision — F1 over true/false positives and negatives at ambiguity nodes 3. Interaction Strategy Rationality — is choosing search/clarify/answer appropriate at each node? 4. Cost Efficiency — balance of interaction turns, search API calls, and clarification requests

A user simulator provides progressive cues consistent with real user behavior, including partial/incomplete answers, allowing follow-up questions — closer to real interaction than one-shot full answers.

Key Findings

• Proactive clarification > blind searching: task success rates are significantly higher when agents ask at ambiguity nodes
• Even strong models (GPT-4, Claude families) frequently choose "keep searching" over "ask the user"
• Mainstream models correctly identify only ~30–40% of the 463 ambiguity nodes
• Common failure modes: searching when clarification was needed (most dangerous — errors compound), over-clarifying when direct answers suffice, or answering with unresolved ambiguity
• Non-clarifying agents use more search turns yet achieve lower success rates; clarification-aware agents can be more efficient overall

Why models can't "ask": training data bias toward well-defined queries, missing evaluation signals, action spaces lacking a "clarify" action, and overconfidence in their own interpretations.

Why It Matters

1. From information retrieval to conversational retrieval — mirrors how human researchers actually work 2. Deployment necessity — enterprises fear confident wrong answers; existing benchmarks report ~90% accuracy while users complain about irrelevant answers 3. Reusable evaluation methodology — four dimensions beat single accuracy metrics 4. Framework design implications — add a "clarify" action, ambiguity detection modules, and clarification-default decision policies at ambiguity nodes

Limitations and Future Work

• Primarily Chinese-language data; cross-lingual benchmarks needed
• Limited ambiguity taxonomy — metaphor, cultural, and domain-jargon ambiguity remain
• User simulator limits — real users may be vague or wrong; handling confused users is harder
• Multi-agent ambiguity propagation across agents is an open problem
• From evaluation to training: fine-tuning on DiscoBench feedback, ambiguity-aware reward signals in RLHF, and new loss functions encouraging clarification under uncertainty

Bottom Line

A search agent's biggest failure is not "can't find the answer" but "doesn't know what it's asking." DiscoBench shows that modeling ambiguity as a dynamic, propagating phenomenon — and asking one well-placed question — substantially outperforms searching blindly to the end, marking a key step from one-way retrieval tools toward conversational research assistants.

Reference

• Paper: When Search Agents Should Ask: DiscoBench for Clarification-Aware Deep Search (arXiv:2606.27669)
• Dataset: 11 domains, 211 samples, 463 ambiguity instances, 4 ambiguity types
• Framework: task success rate, ambiguity detection precision, interaction strategy rationality, cost efficiency