Honglin Guo, Qi Zhang, Yu Zhang, Weijie Li, Rui Zheng, Zhikai Lei, Qiyuan Peng, Zhiheng Xi, Tao Gui, Qi Zhang
5 min
As large language models (LLMs) transition from standalone chatbots to autonomous agents, their ability to perform reliable knowledge work within internal enterprise archives becomes critical. The authors investigate whether current models can effectively navigate large, disorganized document collections to locate sparse evidence, reconcile conflicting information (such as varying units or terminology), and perform accurate multi-step computations. To measure this, they introduce AGORA, a benchmark designed to test agentic reasoning in realistic, archive-grounded settings.
AGORA consists of 362 natural-language questions paired with 9,664 authentic documents across eight professional domains, including finance, law, healthcare, and technology. The total corpus size of 372 million tokens exceeds the context window of current models, forcing agents to perform deliberate exploration rather than exhaustive scanning. The authors developed an agentic pipeline to construct the benchmark, which includes cross-document task synthesis, leakage-preventing obfuscation, and rigorous difficulty filtering to ensure tasks cannot be solved via parametric knowledge alone. Evaluation is conducted using a minimal bash-based harness, allowing agents to interact with the file system to retrieve and analyze evidence before submitting a verifiable numeric answer.
Evaluating eight frontier proprietary and open-weight models, the authors find that archive-grounded document reasoning is far from solved. Even the top-performing model achieved only 59.4% accuracy, indicating significant room for improvement. The results highlight that model performance is highly sensitive to the specific domain, with rankings often shifting across different fields. This suggests that current agents lack the robust, generalizable reasoning skills required to handle the diverse, inconsistent, and large-scale nature of real-world workplace archives.
This study demonstrates that while LLMs show promise as workplace assistants, they currently struggle with the fundamental requirements of enterprise knowledge work: navigating massive, messy file systems and reconciling disparate data. By providing a reproducible, verifiable, and multi-domain benchmark, AGORA offers a necessary tool for researchers to measure and improve the reliability of agentic systems in professional environments.
ls, cat, or grep) to locate and extract information from the document archive.Large language models are increasingly deployed as agents that reason over documents rather than answer from parametric knowledge. We study archive-grounded reasoning: locating sparse evidence across a large, messy collection of workplace files, reconciling inconsistent terminology, units, and time conventions, and computing an answer. Existing benchmarks address only parts of this setting and none jointly stresses archive-groundedness, agentic exploration, and cross-domain coverage. We introduce Agora, a benchmark pairing 362 questions with eight domain collections of 9,664 authentic documents and 372M tokens, far exceeding any model's context window, so agents must explore deliberately rather than scan exhaustively. Agora is built by an agentic pipeline combining cross-document task synthesis, leakage-preventing obfuscation, and difficulty filtering. Evaluating eight models, we find the task far from solved: even the strongest reaches only 59.4% accuracy, with notable variation across domains.
Sam: Right. And it's not just finding the file. It's about synthesis. The agent might find a dollar figure in one report expressed in one currency, and a related figure in a spreadsheet using a different unit. It has to recognize that discrepancy, convert correctly, and arrive at a single coherent answer. That's the kind of multi-step reasoning the benchmark is designed to stress-test.
Alex: And how do they verify the agent is actually reasoning rather than just making an educated guess?
Sam: Every task in the benchmark has a single, verifiable numeric answer. You can't fake a math result. The researchers built a multi-stage pipeline: gather documents, construct questions that genuinely require consulting multiple files, then filter out any question that turns out to be too easy or answerable from a single source. The goal is to ensure each question is a real test of connected reasoning, not just lookup.
Alex: So how are current models actually performing on this?
Sam: The results suggest we're still in the early stages. Even the strongest model tested reached only around 59% accuracy. That's a meaningful gap. It tells us that while these agents can handle straightforward retrieval tasks, they still struggle when the answer requires connecting scattered, sometimes contradictory pieces of information across a large archive.
Alex: So the bottleneck isn't raw intelligence in some general sense — it's more like the ability to stay organized and disciplined while searching through a haystack.
Sam: That's a fair way to put it. And the paper adds another layer: performance varies significantly depending on the domain. Finance questions and legal questions both live in the same archive, but they have very different structures, vocabularies, and conventions. A model that's reasonably good at one might struggle with the other. It's a reminder that "intelligence" in this context is always relative to the specific landscape the agent is navigating.
Alex: There's something worth flagging here, though. If the researchers used AI models to help filter which questions were difficult enough, doesn't that risk biasing the benchmark against those same models?
Sam: That's a genuine limitation the paper acknowledges. Because some of the models used in the filtering process are also subjects in the evaluation, there's a real risk the benchmark ends up being inadvertently calibrated around their specific weaknesses. The researchers tried to minimize this, but they admit that a fully independent human review panel would produce a more robust result. It's an honest caveat.
Alex: So where does this lead? If agents eventually get reliable at this kind of work, what does that actually unlock?
Sam: The vision is to move beyond chatbots toward something closer to an autonomous research assistant — a system you could hand a company's entire document archive to and ask a complex question, and it would come back with a sourced, reasoned answer rather than a plausible-sounding guess. AGORA is an attempt to define what "good" looks like for that kind of system, so researchers have a shared target to work toward. Whether current models can close that 41% gap is the open question this benchmark is designed to keep asking.
Alex: That's a useful frame. Not "is AI smart?" but "can AI do the specific, disciplined work of a careful researcher?" Thanks for walking us through it, Sam — and thanks to everyone listening to ResearchPod.