Enterprise workflows increasingly rely on agents for \emph{schema-guided extraction}: given a document and a user-defined schema, the agent faithfully follows the schema to produce the correct output with source evidence as grounding metadata. We present ExtractBench, a benchmark for schema-guided extraction and, to our knowledge, the first to score value accuracy, record completeness at scale, grounding, and measured cost together. The evaluation system contains 4,869 pages across 370 enterprise documents, 8 business domains, and 67 document types, with clear tags differentiating their challenge scenarios. The scalable schema and ground-truth curation pipeline combines independent-system agreement for real documents, known values for synthetic lists, and human verification for forms. We report order-insensitive value F1 for value accuracy, plus two grounding metrics for source traceability: word- and page-level F1. Commercial VLMs perform well on short documents but often truncate record lists on long ones, while coding agents retain higher accuracy at much higher cost. LlamaExtract Agentic Plus ranks first on all three metrics, with accuracy comparable to coding agents at a fraction of the cost. Dataset and evaluation code are available on \href{https://huggingface.co/datasets/llamaindex/ExtractBench}{HuggingFace} and \href{https://github.com/run-llama/ExtractBench}{GitHub}.
Alex: Welcome to another episode of ResearchPod. Today we're looking at a study called ExtractBench, which examines how well AI systems can pull specific data out of complex business documents.
Sam: So this is basically asking whether AI is actually ready to replace the people who manually type information from forms into databases?
Alex: That's the core question. Think about what those workers do every day — they open a hundred-page insurance filing, find the right numbers buried in dense tables, and type them accurately into a system. AI can handle simple documents reasonably well, but this research is asking what happens when the documents get long, messy, or complicated.
Sam: And I imagine for a bank or an insurance company, even one missed line in a hundred-page document is a serious problem.
Alex: Exactly. The researchers found that most AI models struggle with what they call "long-list completeness." Picture a document with forty line items in a table. The AI might handle the first ten perfectly, then quietly stop — it just truncates the rest, as if the page ended early. The system doesn't flag an error. It simply leaves data behind.
Sam: That's a subtle failure. You wouldn't even know something was missing unless you checked carefully. So how do the researchers actually measure that?
Alex: They built what they call a "Challenge-Tagged Taxonomy." Here's a useful way to think about it: instead of giving a patient one overall health score, a doctor runs specific tests — checking for fever, checking blood pressure, checking for infection. Each test targets a different problem. This benchmark does the same thing for AI. It tags each document with the specific difficulty it contains — whether that's scan noise, a complicated table layout, or simply a very long list — and then checks whether the AI fails on that specific challenge.
Sam: So rather than one big score that hides the details, you can see exactly where the system breaks down. A model might be fine with noisy scans but fall apart on long lists.
Alex: Precisely. And that distinction matters enormously in practice. If you know your documents are mostly clean but very long, you need a different tool than if your documents are short but photographed at an angle.
Sam: That makes sense. But how do they establish the correct answers in the first place? How do they know what the AI should have found?
Alex: This is where it gets careful. For real-world documents, they run several different AI models on the same document and then have human reviewers step in wherever the models disagree. No single model is trusted to be the judge. For very long lists — where human checking becomes impractical — they generate synthetic data, meaning they build documents from scratch where every answer is already known by design. It's a hybrid approach that combines human oversight with programmatic verification.
Sam: So they're not just asking one AI to grade another AI's homework. They're using disagreement between multiple systems as a signal that a human needs to look more closely.
Alex: Right. And there's one more layer: something called "visual grounding." The AI isn't just asked to produce an answer — it must also point to the exact location on the page where it found that answer. A specific cell in a table, a particular line of text.
Sam: Why is that so hard? If the AI found the number, shouldn't it know where it came from?
Alex: You'd think so, but consider what the document actually looks like. Imagine a table with three hundred rows of nearly identical entries — same formatting, same column structure, slightly different numbers. Finding the right row and the right cell, and then linking your answer back to that precise location, requires a level of precision that current systems often lack. The AI might extract the correct value but from the wrong row, or it might not be able to retrace its steps at all. For industries like law or finance, where you need a complete audit trail, that's a meaningful gap.
Sam: So we're moving from asking "did the AI get the right answer?" to asking "can the AI show its work?" That's a much higher bar.
Alex: It is. And without that transparency, the people using these systems are essentially taking the AI's word for it — which isn't acceptable when the stakes are high.
Sam: So what does the benchmark actually reveal about which approaches hold up best?
Alex: The study suggests a real trade-off. Commercial AI models — the kind you can access through an API — tend to be fast and relatively inexpensive, but they're the ones most likely to truncate long lists or lose precision on complex tables. Specialized agents that can actually write and execute code to parse a table tend to be more consistent on difficult documents, but they're slower and cost more to run.
Sam: So it's not simply a question of which model is more capable in the abstract. It's about matching the right tool to the actual documents you're dealing with. A simple receipt is a very different problem from a fifty-page regulatory filing.
Alex: That's the central insight. The researchers argue that chasing a higher overall score on a benchmark can actually be misleading. What matters is understanding the specific failure modes — truncation, noise sensitivity, grounding errors — and knowing which ones your real-world workflow can tolerate and which ones it cannot. Reliability, in this context, means being able to diagnose what went wrong, not just knowing that something did.
Sam: That feels like a useful reframe for anyone evaluating these tools. The question isn't just "how accurate is it on average?" but "where exactly does it fail, and does that failure matter for my use case?"
Alex: Exactly. And ExtractBench is designed to make that question answerable. Thanks for listening to ResearchPod.