Ismaël Rousseau, Géraldine Damnati, Frédéric Béchet
8 min
Search agents, which combine large language models with iterative retrieval tools, are increasingly used for complex, multi-step question answering. However, standard benchmarks often fail to capture how these agents perform when faced with low-quality or misleading information. DeepStress addresses this by providing a controlled laboratory environment that replaces a live search engine with a synthetic document generator. This allows researchers to systematically degrade evidence quality along three axes: trustworthiness, relevance, and factuality.
DeepStress intercepts an agent's search calls and returns synthetic documents tailored to the agent's specific query. By manipulating the prompt instructions for these documents, the authors can control the probability that a retrieved document is trustworthy, relevant, or factually correct. This setup enables researchers to observe how agents react to "noise" in real-time, such as whether they detect contradictions, abstain from answering when evidence is poor, or blindly incorporate incorrect information into their final response.
The study evaluates twelve different search agents, including various GPT-based models and open-weight systems like DR-Tulu and Search-R1. The results demonstrate that agents exhibit vastly different failure modes: some models are prone to confident hallucinations when provided with incorrect evidence, while others are more likely to abstain or exhaust their tool-call budget. To quantify these behaviors, the authors introduce the Reliability-Aware Score (RAS), which rewards agents for correct answers and appropriate abstentions under low-quality conditions, while penalizing them for incorrect answers. This metric provides a more nuanced view of agent robustness than traditional accuracy scores.
As search agents move from research prototypes to real-world applications, understanding their failure modes under adverse conditions is critical. DeepStress provides a standardized way to stress-test these systems, helping developers identify whether their agents are truly robust or merely lucky in their retrieval environment. By decoupling the retriever from the reasoning model, this framework allows for a clearer analysis of how agents manage conflicting information and when they should decide to stop searching.
While search agents demonstrate impressive capabilities in multi-step question answering, their robustness to poor-quality evidence remains under-explored. This phenomenon occurs rarely in realistic benchmarks but can lead to dramatic failure in real life applications. Therefore in this study we propose DeepStress, a stress testing framework that controls the frequency of challenging evidence by replacing the retrieval module of search agents with a controlled synthetic environment. We use this framework to control three dimensions that can affect document reliability: trustworthiness, relevance, and factuality. Testing several search agents on HotpotQA and BrowseCompPlus, we demonstrate that agents exhibit substantial differences in their ability to handle unreliable information and propose new metrics that better document systems outcomes as well as the interactions between conflicting parametric and retrieved knowledge.
Alex: Can you make that separation tangible without treating a credible-looking page as proof of truth?
Sam: Their examples use a query about Debussy’s orchestral innovations. One generated passage gives a sober description of his use of instrumental colour. Another delivers factual content in sensationalist, clickbait language. That separates surface credibility from correctness: the presentation can look dubious without the underlying information being false.
Alex: And how do they control correctness itself? If the generator invents everything freely, the labels could become wishful thinking.
Sam: For intended factual documents, generation uses supporting evidence supplied by the question-answering dataset. For intended false documents, the generator is instructed to introduce severe factual errors. Irrelevant results instead draw on another question’s supporting material. The quality settings determine probabilities, so a mixed condition does not guarantee an exact balance in every run.
Alex: Before we interpret the failures, how broad was the actual evaluation?
Sam: They evaluated twelve systems, covering closed-weight models and several open-weight search agents. Across both datasets and the different evidence conditions, they recorded about thirty-four thousand runs. One dataset was HotpotQA, which combines facts across Wikipedia passages. The other was BrowseComp-Plus, with harder searches constrained by multiple clues.
Alex: What contrast best shows why a single accuracy score would miss the point?
Sam: Consider Search-R1. Making sources look less credible barely changed its correctness, which stayed around four in five answers. But when results became entirely irrelevant, its confident wrong answers rose to roughly four in five. Those are different outcomes under different degradations, not a general story of robustness.
Alex: So ignoring source appearance can look harmless when the information remains useful. It becomes dangerous to infer that the same agent can handle evidence that is useless or false.
Sam: The GPT-based systems showed another pattern. Under degraded evidence, they more often abstained or kept searching until they exhausted their allowance. Lower reasoning effort tended toward abstention; higher effort tended toward prolonged searching. Some other systems increasingly failed to produce the required closing answer marker, which the study counted separately.
Alex: A relevant falsehood sounds harder to reject than a passage about the wrong topic. Did the experiments support that distinction?
Sam: For some systems, yes. SpiqaDR abstained less with entirely false evidence than with entirely irrelevant evidence. DR-Tulu also produced more confident errors under factuality degradation than relevance degradation. The authors interpret that pattern as suggesting that relevant false documents are harder to reject, not as proving it for every agent.
Alex: You also flagged correct answers that might not demonstrate successful search. How did they separate evidence use from what a model already knows?
Sam: They compared the datasets when every relevant result contained factual errors. On HotpotQA, GPT-5-mini at medium reasoning effort still answered sixty percent correctly. On BrowseComp-Plus, correctness fell almost to zero. They also removed search entirely; that baseline showed the same broad contrast between datasets.
Alex: That suggests HotpotQA lets agents fall back on parametric knowledge—the information stored in their learned parameters. A correct answer there does not necessarily validate their handling of retrieved evidence.
Sam: The authors therefore argue that HotpotQA evaluations need a no-search baseline. The mixed-factuality condition offers another useful test. Correct documents share supporting evidence, while false documents are not designed to agree. Repeated searches can therefore, in principle, reveal a stable signal amid conflicting claims.
Alex: Now the scoring objection: avoiding a wrong answer is useful, but searching forever is not. How does their new metric handle that?
Sam: Their Reliability-Aware Score gives full credit for correctness and none for wrong or malformed answers. Abstention and exhausted search budgets receive credit that increases as the evidence condition becomes less reliable. At a completely unreliable extreme, that credit can reach the same level as a correct answer.
Alex: So it encodes a preference, rather than discovering an objective definition of good behavior. Does the document acknowledge that distinction?
Sam: It does. The authors discuss giving budget exhaustion less credit than explicit abstention. They also introduce a cost-normalized version, which penalizes expensive searching. Their cost combines input, output, and search calls using chosen weights. Both metrics are summaries, not substitutes for the detailed outcome categories.
Alex: How much confidence should we place in the simulator’s labels, and how far beyond this laboratory do the results travel?
Sam: Humans checked fifty generated documents, with factuality labels matching their judgments about eighty-six percent of the time. That supports the manipulation, but shows it is imperfect. More fundamentally, results contain only one synthetic document per call, and the experiments do not combine degradations. Real search can deliver multiple conflicting, incomplete results together; more complex agent pipelines were not tested.
Alex: Given those boundaries, who should read the full document, and where should they begin?
Sam: Researchers building or evaluating search agents should start with the dataset breakdown, especially the false-evidence and no-search comparisons. Then read the metrics discussion and limitations before adopting the proposed score. For everyone else, remember this: accuracy alone cannot tell cautious failure from confident misinformation.
Alex: Thanks for listening.