Abhigya Verma, Khyati Mahajan, Amit Kumar Saha, Shruthan Radhakrishna, Sagar Davasam, Vikas Yadav, Sai Rajeswar
5 min
Abstract
Vision language models (VLMs) have achieved strong performance on visual document understanding benchmarks such as DocVQA, ChartQA, and MMLongBench-Doc. However, real-world documents combine multiple factors such as length, layout complexity, modality, and question difficulty, which makes it difficult to attribute model failures to specific causes. We introduce SynthDocBench, a fully synthetic benchmark for long-context visual document understanding that systematically controls factors including document length, layout structure, modality composition, and question type. The benchmark is constructed using a combinatorial design, each factor is varied independently across generated documents, enabling controlled analysis of model behavior. Documents are generated end to end using an LLM pipeline across six layout archetypes, with a 40 percent random override to prevent models from exploiting spurious correlations. Additionally, SynthDocBench spans long-context documents with substantially greater length and structural diversity than existing benchmarks. Evaluating seven frontier VLMs, we uncover three failure modes that existing benchmarks cannot surface: sharp degradation with document length, a systematic positional sensitivity in which the middle third of a document is hardest for five of six models and five of six models show a negative Early-to-Late trend (steepest decline: 8.3 percentage points), and breakdown of chart comprehension in long-document settings. These results suggest that current models may be overfitting to benchmark artifacts rather than achieving robust long-context visual document understanding.
Sam: Wait — so it's not just about length. It's about where in the document the information sits?
Alex: Right. Think of it like how you remember a story. You tend to hold onto the beginning and the ending more clearly than the middle chapters. These models show a similar pattern — the opening and closing sections of a document anchor their attention, and the middle gets crowded out.
Sam: And the third finding?
Alex: Chart comprehension breaks down in context. Models that perform well when you show them a chart in isolation — just the image, nothing else — start failing when that same chart is embedded inside a long report. The surrounding text and pages seem to interfere with their ability to read the visual information accurately.
Sam: That's a meaningful finding. It suggests these models might be doing well on simpler tests not because they've genuinely learned to handle complex documents, but because those tests don't capture what actually happens in the real world.
Alex: That's the concern the paper raises. A model can look capable on a standard benchmark and still struggle with a fifty-page technical report — because the benchmark never tested that combination of length, position, and visual complexity together.
Sam: Is the middle-of-document problem a memory issue? Are the models just "forgetting" what they read earlier?
Alex: It's more of a focus issue. These models process text using a mechanism called attention — essentially, they assign weight to different parts of what they've read, deciding what's relevant to the question at hand. When a document is long, the model's attention tends to anchor on the beginning and end. Information in the middle gets lower weight, even if it's the most relevant part.
Sam: So it's not that the model never processed that information. It's that it doesn't prioritize it when forming an answer.
Alex: That's the working explanation, yes. And it matters practically — because in a real financial report or scientific paper, the critical detail is rarely conveniently placed at the top or bottom.
Sam: So what does this mean for how we should think about AI benchmarks going forward?
Alex: It suggests that a single accuracy score on a standard test can be genuinely misleading. SynthDocBench makes the case that we need diagnostic tools — tests that don't just measure whether a model got the right answer, but that can trace why it got the wrong one. Until that becomes standard practice, we're only seeing a partial picture of what these systems can actually do.
Sam: And a partial picture can give false confidence — which matters a lot if these tools are being used for real technical work.
Alex: Exactly. The paper is a reminder that building capable AI isn't just about training better models. It's also about building better ways to understand where they fall short. Thanks for listening to ResearchPod.