ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction | Boyang Zhang et al. | ResearchPod

Transcript: ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Alex: Welcome to another episode of ResearchPod. Today we're looking at a study called ExtractBench, which examines how well AI systems can pull specific data out of complex business documents.

Sam: So this is basically asking whether AI is actually ready to replace the people who manually type information from forms into databases?

Alex: That's the core question. Think about what those workers do every day — they open a hundred-page insurance filing, find the right numbers buried in dense tables, and type them accurately into a system. AI can handle simple documents reasonably well, but this research is asking what happens when the documents get long, messy, or complicated.

Sam: And I imagine for a bank or an insurance company, even one missed line in a hundred-page document is a serious problem.

Alex: Exactly. The researchers found that most AI models struggle with what they call "long-list completeness." Picture a document with forty line items in a table. The AI might handle the first ten perfectly, then quietly stop — it just truncates the rest, as if the page ended early. The system doesn't flag an error. It simply leaves data behind.

Sam: That's a subtle failure. You wouldn't even know something was missing unless you checked carefully. So how do the researchers actually measure that?

Alex: They built what they call a "Challenge-Tagged Taxonomy." Here's a useful way to think about it: instead of giving a patient one overall health score, a doctor runs specific tests — checking for fever, checking blood pressure, checking for infection. Each test targets a different problem. This benchmark does the same thing for AI. It tags each document with the specific difficulty it contains — whether that's scan noise, a complicated table layout, or simply a very long list — and then checks whether the AI fails on that specific challenge.

Sam: So rather than one big score that hides the details, you can see exactly where the system breaks down. A model might be fine with noisy scans but fall apart on long lists.

Alex: Precisely. And that distinction matters enormously in practice. If you know your documents are mostly clean but very long, you need a different tool than if your documents are short but photographed at an angle.

Sam: That makes sense. But how do they establish the correct answers in the first place? How do they know what the AI should have found?

Alex: This is where it gets careful. For real-world documents, they run several different AI models on the same document and then have human reviewers step in wherever the models disagree. No single model is trusted to be the judge. For very long lists — where human checking becomes impractical — they generate synthetic data, meaning they build documents from scratch where every answer is already known by design. It's a hybrid approach that combines human oversight with programmatic verification.

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Boyang Zhang, Adrian Lyjak, Eli Stewart, Zhaoqi Li, Simon Suo

Jul 31, 2026·5:23·arxiv·Original Paper

Schema-guided extractionExtractBenchGroundingValue F1Enterprise document workflowsSchema-guided document extraction

Listen

5 min

Briefing

Abstract

Enterprise workflows increasingly rely on agents for \emph{schema-guided extraction}: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on \href{https://huggingface.co/datasets/llamaindex/ExtractBench}{HuggingFace} and \href{https://github.com/run-llama/ExtractBench}{GitHub}.

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction: Research Paper Podcast

ExtractBench: A Benchmark for Schema-Guided Enterprise Document Extraction

Listen