Meilin Chen, Hongyuan Bao
8 min
Abstract
Existing Open-Ended Deep Research (OEDR) systems primarily generate reports from scratch, making them inefficient for scenarios where research reports need to be continuously maintained as new information emerges. We introduce \textbf{Incremental Open-Ended Deep Research (Incremental-OEDR)}, a research setting that treats a report as an evolving research state and incrementally updates it by preserving valid knowledge, revising outdated or incomplete content, and incorporating newly available information. To support this setting, we propose \textbf{Structured Harness}, which represents reports as structured collections of outlines, sections, and supporting evidence, and provides structured retrieval, a persistent structured evidence pool, and structured generation for selective report updating and evidence reuse. We further establish a temporal evaluation framework spanning ten years, with \emph{Single-Step Task} and \emph{Long-Chain Task} to evaluate incremental updates over both individual transitions and long-term update chains. Extensive Experiments on DeepResearch Bench and DeepConsult under both the Open-source Configuration (OC) and Proprietary Configuration (PC) show that Incremental-OEDR maintains competitive report quality while substantially improving report continuity and reducing research costs. As shown in Figure~\ref{fig:profile}, it achieves up to 0.51 higher content-level ROUGE-L F1, 0.63 higher outline-level EM F1, 33\% lower token consumption, and 61\% fewer search calls than OEDR on DeepResearch Bench. For more details, please refer to our project page: https://ioedr-project.github.io/.
Sam: Editing the text is the easy part. How does it know which sources have actually changed without re-reading the whole web?
Alex: That's the evidence pool, a stored record of every source the report cited. For each one it keeps the web address, the title, the raw text, and a timestamp. On an update, a tool refetches those pages and computes a cheap character-level similarity against the stored copy. Only if similarity falls below a threshold, set at 0.9, does it call a language model. That model then works out, claim by claim, what changed.
Sam: Give me a concrete case of that working.
Alex: Their own illustration uses OpenAI's announcement page for its deep research product. The stored version includes an earlier update note. Later, the refetched page has a new note saying you can now connect deep research to external tools through a standard protocol. The text has shifted, so the diff step fires and picks out that one addition. Pages that haven't changed pass straight through, no model call needed. New sources found by fresh searches simply get added to the pool for next time.
Sam: So the expensive reasoning only happens where something moved. Now, how do you test this? You can't wait ten years for the web to change.
Alex: They simulate time. They pick December thirty-first of each year from 2016 to 2025. For each date, the search tool only returns pages published before it. There are two tasks. In the single-step task, the system updates a from-scratch report from the previous year once. In the long-chain task, it keeps updating its own output, year after year.
Sam: And what's the comparison?
Alex: The base agent they build on, an open-source system called ms-agent, run fresh at each date. They use two benchmarks. The main one, DeepResearch Bench, has 100 expert-written research tasks across 22 fields. The other, DeepConsult, focuses on business and consulting questions. And they test both an open-source model setup and one with a proprietary model as the lead agent.
Sam: So, the results. Does quality suffer when you edit instead of rewrite?
Alex: Not by their measure. On DeepResearch Bench with the open-source setup, the from-scratch agent averages about 49 on the benchmark's quality score. The incremental version averages about 49 too, slightly higher in both tasks. I'd call that comparable, not better. The gaps are fractions of a point, and the score comes from a language model acting as judge.
Sam: Then where's the real gain?
Alex: Continuity and cost. They measure text overlap between consecutive versions of a report, using a score called ROUGE-L. For fresh reruns, content overlap is about 0.17. For the long chain of incremental updates, it's about 0.68. At the outline level the contrast is starker. Exact heading matches go from almost none, 0.04, to about two thirds.
Sam: To be fair, high overlap is partly the design goal. A system that never changed anything would score perfectly.
Alex: That's why quality has to be read alongside it. Overlap alone doesn't prove the updates were good. The paper's case is that overlap went up while the quality score held steady.
Sam: And in money terms?
Alex: In the long chain, total tokens per report fall by about a third. Search calls drop by about 61 percent. The proprietary setup and DeepConsult show the same pattern.
Sam: Why would the long chain be cheaper than a single update? I'd have guessed errors pile up over the chain.
Alex: It's the evidence pool. The single-step task starts from a from-scratch report, which has no stored evidence pool. The long chain inherits a full pool from the previous update, so it only refreshes what changed. One caveat from the numbers themselves: on DeepConsult, searches drop in the long chain but tokens are essentially flat, even slightly up.
Sam: Does how often you update matter?
Alex: They tested intervals from ten years down to one month. Ten-year gaps were worse than rerunning: about one and a half times the cost, with slightly lower quality. Gains leveled off around one year, so that's what they use throughout.
Sam: What's the weakest link in all this?
Alex: The time simulation. Restricting search by publication date controls which pages are visible. It can't recreate how an existing page's content looked in, say, 2018. So the refresh mechanism isn't fully exercised by the historical setup. They also note the benchmarks weren't designed for incremental maintenance. And I'd add that everything is built on one base agent.
Sam: One side finding seemed bigger than the method to me.
Alex: The same system scores noticeably better at later dates, simply because more information exists. Their point is that judging these agents only on today's web may not fully characterize them.
Sam: So who should read the full paper, and where should they start?
Alex: Anyone building research agents or pipelines that maintain reports over time. Start with the section on the Structured Evidence Pool, then the analysis of why the long chain costs less than a single step, then the update-interval study. The appendix tool list shows exactly what the agent can call.
Sam: And for everyone else?
Alex: A research report doesn't have to be rewritten to stay current. Treat it as a living document, check what changed, and edit only that.
Sam: Edit, don't regenerate. Thanks for listening.