Yiling Tao, Shihan Deng, Meiling Tao, Pengzhi Wei, Zhichao Hu, Zhihao Zhu
4 min
As search agents transition from static retrieval to autonomous, multi-step reasoning, they often encounter vague or underspecified queries. This paper investigates whether these agents can proactively identify ambiguity, engage in effective multi-turn clarification, and recover correct reasoning paths, or if they instead propagate errors by blindly continuing their search.
The authors introduce DISCOBENCH, a benchmark designed to evaluate interactive, clarification-aware deep search. Unlike static benchmarks, DISCOBENCH models ambiguity as a dynamic phenomenon that emerges during multi-step reasoning. It contains 211 samples and 463 ambiguity instances across 11 domains, covering four types: Entity, Version, Criteria, and Factual Inaccuracy. The authors also developed a user simulator that provides discriminative clues when an agent successfully asks for clarification, allowing for a standardized evaluation of task utility, detection accuracy, and interaction strategy.
Experiments on representative LLMs reveal a significant gap between retrieval capability and interactive problem-solving. The study finds that ambiguity detection and the ability to formulate effective clarification questions are distinct, often underdeveloped skills. A critical failure mode identified is that many models prefer to continue searching—even when faced with ambiguous nodes—rather than asking the user for help. This behavior frequently leads to lower success rates than simply guessing, suggesting that current agents lack the meta-cognitive awareness to recognize when their retrieval path has become unreliable due to underspecified input.
This research highlights that for search agents to move beyond simple retrieval and become truly autonomous assistants, they must master the art of conversation. The findings suggest that future development should focus on integrating uncertainty estimation with interactive strategies, ensuring that agents can pause and collaborate with users when the path forward is unclear, rather than wasting computational resources on incorrect reasoning chains.
Search agents powered by large language models (LLMs) are increasingly used to solve complex information-seeking tasks, requiring multi-step retrieval and reasoning to fulfill user goals. However, existing benchmarks often assume that user queries are complete and explicit, overlooking the fact that real-world search requests are frequently vague, underspecified, or even factually incorrect. In deep search scenarios, such ambiguity can propagate along multi-step reasoning chains and lead agents toward incorrect search trajectories. To address this gap, we introduce DiscoBench, a benchmark for clarification-aware deep search, designed to evaluate whether search agents can proactively identify ambiguity, ask effective clarification questions, and recover correct reasoning paths through user interaction. DiscoBench contains 211 samples and 463 ambiguity instances across 11 real-world domains, covering four ambiguity types. We further design a user simulator for multi-turn interaction and evaluate model performance from four perspectives: task utility, ambiguity detection, interaction strategy, and cost efficiency. Experiments on representative LLMs show that ambiguity detection and effective clarification are distinct capabilities, and that repeatedly searching instead of asking for clarification often performs worse than direct guessing, highlighting a critical gap between retrieval ability and interactive problem-solving in current search agents.
Sam: So doing more work, but in the wrong direction, is actually more harmful than doing nothing?
Alex: That's what the evidence points to. Each additional search step taken under a false assumption doesn't correct the error—it reinforces it. The agent finds information that seems to confirm its wrong interpretation, and the gap between what it's doing and what the user actually wanted grows wider with every step.
Sam: So the problem isn't that these agents are bad at searching. It's that they're bad at knowing when they shouldn't search yet.
Alex: And the paper draws a clear distinction here. Recognizing that a question is ambiguous is one skill. Knowing how to ask a useful, targeted question to resolve that ambiguity is a separate skill entirely. The research suggests that many current models lack both, and that closing that gap is where the meaningful work lies.
Sam: I'll think differently about AI search tools after this. When one gives me a confident-sounding answer to something I asked vaguely, I'll wonder whether it actually understood me—or just committed to a guess.
Alex: That's a reasonable instinct. The authors' broader point is that for these systems to become genuinely reliable, they need to learn something that good human communicators do naturally: pause, check understanding, and only then proceed. Thanks for listening to ResearchPod.