Lei Xiong, Huaying Yuan, Zheng Liu, Zhao Cao, Zhicheng Dou
9 min
PaperScope is a groundbreaking benchmark designed to evaluate 'agentic deep research' systems—AI agents that autonomously handle complex scientific workflows like retrieving papers, reasoning across documents, and generating insights. Unlike traditional benchmarks that test single documents or web browsing, PaperScope focuses on multi-modal, multi-document scientific reasoning, where agents must integrate text, figures, tables, formulas, and algorithms from hundreds of papers. Built on over 2,000 AI papers from top conferences (spanning 202–1500 documents per task), it uses a knowledge graph for structured grounding, ensuring questions have dense, semantically rich evidence scattered across sources.
This addresses a critical gap: real science involves synthesizing multi-source info (e.g., comparing algorithms from RL papers with figures from vision papers), but existing tools like RAG struggle with long contexts and modalities. PaperScope reveals even top agents (OpenAI Deep Research, Tongyi) score poorly, highlighting needs in retrieval, multi-hop reasoning, and multimodal integration.
PaperScope starts with a knowledge graph (KG) from top AI conferences, using an 'inverted-construction' strategy: random walks sample thematically coherent paper sets, creating high-quality, near-unique QA pairs grounded in the graph. This ensures tasks mimic real research—e.g., 'How do recent RL methods compare visually to CV baselines?'
Questions demand cross-document synthesis: answers hide in figures/tables from paper A, text from B, formulas from C. The corpus covers AI subfields (RL, generative models, CV), with 25k+ ArXiv/OpenReview papers for scale.
Over 2,400 QA pairs across 4 meta-categories and 11 sub-tasks:
Tests on 16 ReAct-based agents show major limitations: low accuracy in multimodal understanding (e.g., chart reasoning), retrieval (long-context failure), and integration. Table 1 compares PaperScope favorably to priors—superior in multi-doc/pages (avg. 202+), modalities, reasoning depth.
Agentic systems promise to accelerate science but falter on 'deep research' without benchmarks like this. PaperScope provides a scalable pipeline for datasets, pushing agents toward human-like research (planning, tool use, dynamic refinement). For students: think of it as evolving benchmarks from 'read one paper' to 'conduct literature review autonomously.'
Available at: https://github.com/CherYou/PaperScope.
Leveraging Multi-modal Large Language Models (MLLMs) to accelerate frontier scientific research is promising, yet how to rigorously evaluate such systems remains unclear. Existing benchmarks mainly focus on single-document understanding, whereas real scientific workflows require integrating evidence from multiple papers, including their text, tables, and figures. As a result, multi-modal, multi-document scientific reasoning remains underexplored and lacks systematic evaluation. To address this gap, we introduce PaperScope, a multi-modal multi-document benchmark designed for agentic deep research. PaperScope presents three advantages: (1) Structured scientific grounding. It is built on a knowledge graph of over 2,000 AI papers spanning three years, providing a structured foundation for research-oriented queries. (2) Semantically dense evidence construction. It integrates semantically related key information nodes and employs optimized random-walk article selector to sample thematically coherent paper sets, thereby ensuring adequate semantic density and task complexity. (3) Multi-task evaluation of scientific reasoning. It contains over 2,000 QA pairs across reasoning, retrieval, summarization, and problem solving, enabling evaluation of multi-step scientific reasoning. Experimental results show that even advanced systems such as OpenAI Deep Research and Tongyi Deep Research achieve limited scores on PaperScope, highlighting the difficulty of long-context retrieval and deep multi-source reasoning. PaperScope thus provides a rigorous benchmark alongside a scalable pipeline for constructing large-scale multi-modal, multi-source deep research datasets.
Alex: That sounds efficient for spotting similarities. So once the big merged graph is ready, how do they pick groups of papers that force cross-checking?
Sam: They run something called ORWAS, which does biased random walks on this graph—starting from certain nodes and strolling along edges, but favoring busy hubs like popular datasets or methods to stay on-topic. It's optimized for speed with tricks like pre-caching neighbors and batching walks, ending up with sets of papers that share key nodes, ensuring questions need evidence from multiple docs. Picture wandering a subway: you bias toward transfers at crowded stations to hit linked lines, guaranteeing clusters that demand switching trains—or papers—to solve the puzzle.
Alex: Huh. So the walks create these dense, connected bundles on purpose. Does that mean the questions are built straight from those shared spots?
Sam: Yes—these bundles feed into generating questions using an inverted approach: start with known facts across the papers to craft unique answers verifiable right there, no outside guessing needed. They also lock key clues into visuals like figures or tables only, blocking text shortcuts, and filter for hardness by checking if top models fail them. This setup reveals real limits, as even leaders like Grok-4 hit just around 40% on reasoning, about twice some baselines but still far from reliable multi-doc synthesis.
Alex: Right, so retrieval helps, but the bottleneck's in chaining visuals and logic across papers. That explains why web search flops in their tests—they grab wrong docs.
Sam: Exactly. Local graph-guided retrieval beats generic searches here, underscoring how semantic density from ORWAS drives the challenge.
Alex: If ORWAS creates these dense bundles efficiently, how did they figure out the right settings for the walks—like how long each stroll should be or how strongly to bias toward popular spots?
Sam: They tested different tweaks, called hyperparameters, starting with walk length—the number of steps in each path. Short paths trap the walk near the starting facts and miss broader links, like staying in one neighborhood instead of exploring the city. Medium ones at around 100 steps strike a balance, hitting the most unique combinations without wandering too far and picking up unrelated noise.
Alex: Right, so not too short to be boring, not too long to get lost. What about biasing toward those busy hubs—did that have a sweet spot too?
Sam: Yes, the bias probability controls how often to jump to popular nodes during walks from papers. A balanced setting around 0.3 lets it explore rarer facts without ignoring common ones, yielding the widest spread of paper groups. Too high sticks to hubs and repeats the same clusters, cutting diversity sharply.
Alex: That explains the speed you mentioned earlier. Does this setup hold up outside AI papers, or is it tuned just for that?
Sam: They checked with a small test on 24 papers from medicine and mechanics fields. By tweaking the info types slightly—like adding sections on limits for health papers or equations for physics—the graphs built dense networks, and experts rated the resulting questions highly satisfactory. It suggests the method transfers well with minor changes, no heavy retooling needed.
Alex: Huh, promising for other sciences. But there must be some catches to this whole pipeline.
Sam: Indeed, the paper notes key limits. It's built on a fixed set of papers, not endless web sources, so it skips issues like junk retrieval from huge collections. The walks use practical shortcuts that boost speed but aren't perfect global picks, and question-making relies on language models, which can add small biases or noise. Still, this reveals meaningful gaps in agents, pointing a clear path for better multi-doc tools.
Alex: Those limits make sense for a focused test. How did they make sure the questions were actually hard and required looking at pictures and tables, not just skimming text?
Sam: They used a careful building process where experts—graduate students in computer science—picked solid facts from visuals across papers first, like numbers in charts or lines showing trends. Then they wrote questions backward from those facts, so answers are unique to that exact group of papers and can't be guessed from outside info. This "inverted construction" ensures no loose ends, and they forced key clues into images only, rejecting any question solvable from text alone—what they call multi-modal dependence.
Alex: So starting from the answer makes it verifiable. Did humans check everything, or was it all automated?
Sam: For tougher reasoning questions, experts hand-crafted them with rules like needing multiple steps of logic across visuals. Simpler tasks got machine drafts, then expert checks for theme fit and no made-up details. To trust their auto-scoring with big language models as judges, they compared it to three PhD students on 120 examples—strong matches, like a correlation over 0.6, which means solid agreement on logic and quality.
Alex: Huh, that's reassuring for scaling. What happens if you strip out the visuals—do models really need them?
Sam: They tested by hiding images and charts, leaving just scanned text on 100 hard questions. Scores dropped sharply across top models—even ones like Grok-4 went from usable to near zero—proving most puzzles break without seeing the pictures, as clues like table rows or graph peaks aren't in words.
Alex: Right, so it's not cheating with text hacks. Even with perfect retrieval, do agents still flop?
Sam: In an "oracle" test, they handed models the exact right papers—no search needed—and performance rose but stayed limited, around double some baselines yet far from full. Web search tools pulled wrong or shallow docs, worse than their graph method. Error breakdowns on failures show visuals trip them first, then reasoning chains, and retrieval misses plus guesses—pinpointing weak spots in seeing, linking, and sticking to evidence.
Alex: That paints a clear picture of the gaps. Pulling it all together, what does this mean for building better research agents?
Sam: The benchmark exposes clear limits in multi-doc synthesis, but its pipeline—graph sampling to prompt-based QA—scales to train agents on vast scientific sets. Future work could tackle extraction via finer visual parsers and reasoning with structured chains, nearing human-level lit reviews. Limitations like the fixed corpus and model noise in generation mean it's a starting point, not open-world proof.
Alex: That's a meaningful step toward agents that truly handle science workflows, gaps and all. Thanks, Sam—this has been a clear look at testing deep research tools.
Sam: My pleasure, Alex. Thanks for listening to ResearchPod.