ResearchPod Summary
Search agents, which combine large language models with iterative retrieval tools, are increasingly used for complex, multi-step question answering. However, standard benchmarks often fail to capture how these agents perform when faced with low-quality or misleading information. DeepStress addresses this by providing a controlled laboratory environment that replaces a live search engine with a synthetic document generator. This allows researchers to systematically degrade evidence quality along three axes: trustworthiness, relevance, and factuality.
DeepStress intercepts an agent's search calls and returns synthetic documents tailored to the agent's specific query. By manipulating the prompt instructions for these documents, the authors can control the probability that a retrieved document is trustworthy, relevant, or factually correct. This setup enables researchers to observe how agents react to "noise" in real-time, such as whether they detect contradictions, abstain from answering when evidence is poor, or blindly incorporate incorrect information into their final response.
The study evaluates twelve different search agents, including various GPT-based models and open-weight systems like DR-Tulu and Search-R1. The results demonstrate that agents exhibit vastly different failure modes: some models are prone to confident hallucinations when provided with incorrect evidence, while others are more likely to abstain or exhaust their tool-call budget. To quantify these behaviors, the authors introduce the Reliability-Aware Score (RAS), which rewards agents for correct answers and appropriate abstentions under low-quality conditions, while penalizing them for incorrect answers. This metric provides a more nuanced view of agent robustness than traditional accuracy scores.
As search agents move from research prototypes to real-world applications, understanding their failure modes under adverse conditions is critical. DeepStress provides a standardized way to stress-test these systems, helping developers identify whether their agents are truly robust or merely lucky in their retrieval environment. By decoupling the retriever from the reasoning model, this framework allows for a clearer analysis of how agents manage conflicting information and when they should decide to stop searching.
Sam: A search agent can fail cautiously, or fail confidently, and accuracy alone cannot tell you which. DeepStress tests that difference by deliberately worsening the evidence agents receive, revealing sharply different responses.
Alex: If I use these systems to gather research evidence, that distinction matters. Is this a test of existing agents, or a proposal for building better ones?
Sam: It is a testing method with experiments, from Ismaël Rousseau, Géraldine Damnati, and Frédéric Béchet, at Orange Research and Aix-Marseille University. Their document, “DeepStress: Stress-Testing Deep Search Agents,” is a preprint under review. It asks what agents do when search stops providing dependable evidence.
Alex: So the audience is anyone evaluating these systems, not just designing the search engine. We need to know whether a wrong answer reflects bad evidence, poor judgment, or something else.
Sam: And whether a correct answer actually required searching. We will unpack the controlled experiment, the contrasting failures it exposes, and why its proposed score needs careful interpretation.
Alex: Start with the gap in existing evaluations. What can this approach reveal that ordinary question-answering benchmarks cannot?
Sam: A search agent alternates reasoning with calls to an external search tool. Standard evaluations often focus on whether its final answer is correct. That hides whether it rejected misleading evidence, guessed from memory, kept searching unsuccessfully, or simply broke its required output format.
Alex: But the authors discuss BrowseComp-Plus, which already uses a fixed collection of supporting and distracting documents. What remains uncontrolled there?
Sam: BrowseComp-Plus helps separate the retriever’s performance from the answering model’s performance. DeepStress isolates a different question: how does the agent respond when evidence quality changes? It intercepts each search call and generates a replacement result, rather than retrieving from that collection or the web.
Alex: Generating results could make the test artificial. Why not prepare a fixed collection of deliberately bad documents?
Because agents issue different queries as their searches unfold. Generating each result on demand lets the simulator respond to the actual query while maintaining the assigned quality conditions. Each call returns a single document. That gives tight control, but also creates the study’s most important realism limitation.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Alex: What does “quality” mean here? A suspicious-looking source and a false statement are not the same problem.
Sam: The authors separate them. Trustworthiness means apparent credibility, conveyed through writing style, page title, and web address. Relevance means addressing the agent’s query. Factuality means whether the document’s claims are correct. The experiments worsen each dimension separately while keeping the others reliable.
Alex: Can you make that separation tangible without treating a credible-looking page as proof of truth?
Sam: Their examples use a query about Debussy’s orchestral innovations. One generated passage gives a sober description of his use of instrumental colour. Another delivers factual content in sensationalist, clickbait language. That separates surface credibility from correctness: the presentation can look dubious without the underlying information being false.
Alex: And how do they control correctness itself? If the generator invents everything freely, the labels could become wishful thinking.
Sam: For intended factual documents, generation uses supporting evidence supplied by the question-answering dataset. For intended false documents, the generator is instructed to introduce severe factual errors. Irrelevant results instead draw on another question’s supporting material. The quality settings determine probabilities, so a mixed condition does not guarantee an exact balance in every run.
Alex: Before we interpret the failures, how broad was the actual evaluation?
Sam: They evaluated twelve systems, covering closed-weight models and several open-weight search agents. Across both datasets and the different evidence conditions, they recorded about thirty-four thousand runs. One dataset was HotpotQA, which combines facts across Wikipedia passages. The other was BrowseComp-Plus, with harder searches constrained by multiple clues.
Alex: What contrast best shows why a single accuracy score would miss the point?
Sam: Consider Search-R1. Making sources look less credible barely changed its correctness, which stayed around four in five answers. But when results became entirely irrelevant, its confident wrong answers rose to roughly four in five. Those are different outcomes under different degradations, not a general story of robustness.
Alex: So ignoring source appearance can look harmless when the information remains useful. It becomes dangerous to infer that the same agent can handle evidence that is useless or false.
Sam: The GPT-based systems showed another pattern. Under degraded evidence, they more often abstained or kept searching until they exhausted their allowance. Lower reasoning effort tended toward abstention; higher effort tended toward prolonged searching. Some other systems increasingly failed to produce the required closing answer marker, which the study counted separately.
Alex: A relevant falsehood sounds harder to reject than a passage about the wrong topic. Did the experiments support that distinction?
Sam: For some systems, yes. SpiqaDR abstained less with entirely false evidence than with entirely irrelevant evidence. DR-Tulu also produced more confident errors under factuality degradation than relevance degradation. The authors interpret that pattern as suggesting that relevant false documents are harder to reject, not as proving it for every agent.
Alex: You also flagged correct answers that might not demonstrate successful search. How did they separate evidence use from what a model already knows?
Sam: They compared the datasets when every relevant result contained factual errors. On HotpotQA, GPT-5-mini at medium reasoning effort still answered sixty percent correctly. On BrowseComp-Plus, correctness fell almost to zero. They also removed search entirely; that baseline showed the same broad contrast between datasets.
Alex: That suggests HotpotQA lets agents fall back on parametric knowledge—the information stored in their learned parameters. A correct answer there does not necessarily validate their handling of retrieved evidence.
Sam: The authors therefore argue that HotpotQA evaluations need a no-search baseline. The mixed-factuality condition offers another useful test. Correct documents share supporting evidence, while false documents are not designed to agree. Repeated searches can therefore, in principle, reveal a stable signal amid conflicting claims.
Alex: Now the scoring objection: avoiding a wrong answer is useful, but searching forever is not. How does their new metric handle that?
Sam: Their Reliability-Aware Score gives full credit for correctness and none for wrong or malformed answers. Abstention and exhausted search budgets receive credit that increases as the evidence condition becomes less reliable. At a completely unreliable extreme, that credit can reach the same level as a correct answer.
Alex: So it encodes a preference, rather than discovering an objective definition of good behavior. Does the document acknowledge that distinction?
Sam: It does. The authors discuss giving budget exhaustion less credit than explicit abstention. They also introduce a cost-normalized version, which penalizes expensive searching. Their cost combines input, output, and search calls using chosen weights. Both metrics are summaries, not substitutes for the detailed outcome categories.
Alex: How much confidence should we place in the simulator’s labels, and how far beyond this laboratory do the results travel?
Sam: Humans checked fifty generated documents, with factuality labels matching their judgments about eighty-six percent of the time. That supports the manipulation, but shows it is imperfect. More fundamentally, results contain only one synthetic document per call, and the experiments do not combine degradations. Real search can deliver multiple conflicting, incomplete results together; more complex agent pipelines were not tested.
Alex: Given those boundaries, who should read the full document, and where should they begin?
Sam: Researchers building or evaluating search agents should start with the dataset breakdown, especially the false-evidence and no-search comparisons. Then read the metrics discussion and limitations before adopting the proposed score. For everyone else, remember this: accuracy alone cannot tell cautious failure from confident misinformation.
Alex: Thanks for listening.