Jennifer D’Souza, Fahad Ahmed, Cecilia Andrea Bustamante Andrade, Lina Frolova, Poorani Gnanasambandan, Dilshad Hussain, Muhammad Uzair Khan, Nkembeng Kevin Nkengfoa, Paul Praveen J., Fabio Priante, Sjoerd Franciscus van der Werf, Thomas Frederik Jan van Roeden
6 min
Abstract
Scientific figures and tables encode essential experimental evidence, yet remain difficult for digital libraries and multimodal AI systems to retrieve and interpret. The ALD/E-ImageMiner benchmark and ICDAR 2026 Competition on Information Extraction from Atomic Layer Deposition/Etching Scientific Figures provide 1,951 figures from 205 publications, expert-annotated for classification, data table extraction, summarization, and visual question answering. In these companion proceedings, we present a forward-looking perspective on how the benchmark can guide future scientific-image challenges. We examine how its tasks probe capabilities from visual and quantitative reading to domain-grounded reasoning and evidential justification, and how Bloom-informed question design can support deeper scientific understanding. We propose "scientific conceptual understanding from images" as a long-term benchmark objective, with future directions including broader domains and figure types, contextual and cross-document synthesis, hypothesis evaluation, provenance, uncertainty, counterfactual grounding, and open-ended multimodal research. This perspective connects the ICDAR 2026 challenge to a broader agenda for machine-actionable scientific visual knowledge and verifiable multimodal scientific AI.
Alex: That last one sounds the most demanding. What kind of questions?
Sam: Think of it like a tiered school exam. The easier questions test basic recall—what's on the x-axis, what value does this bar reach. The harder questions ask the system to reason about what's happening: why does the film thickness change at this temperature, or how does this precursor chemical affect the final material's properties?
Alex: So it's not enough to just spot a shape in the chart. The system has to understand what that shape means in a scientific context.
Sam: Exactly. And the answers come in four formats—yes or no, exact numbers, lists, and full explanatory paragraphs. The paragraph format is particularly revealing, because it forces the model to make its reasoning explicit. You can't hide behind a lucky guess when you have to explain your logic in full sentences.
Alex: It's like the difference between a multiple-choice test and an essay exam. The essay shows whether you actually understood the material.
Sam: That's a good way to put it. And that distinction matters enormously here, because one of the paper's key concerns is hallucination—where an AI produces a confident, fluent answer that is simply wrong. A model can generate a perfectly grammatical paragraph about a chart while describing numbers that don't appear anywhere in the figure.
Alex: How do you even catch that if the answer sounds plausible?
Sam: One approach the authors propose is using controlled counterfactuals. You deliberately change something in the figure—alter a plotted value, remove a data panel—and then ask the same question again. If the model's answer doesn't change to reflect the new evidence, it wasn't actually reading the figure. It was pattern-matching from memory or making things up.
Alex: That's a clever way to probe whether the reasoning is genuine. If the evidence changes, the conclusion should change.
Sam: Exactly. And that kind of test is what separates a system that appears to understand from one that actually does. The authors are fairly direct that current models, even the most capable vision-language systems—AI that processes both images and text together—still fall short on this benchmark in meaningful ways.
Alex: Are there practical barriers beyond just the technical difficulty?
Sam: Yes, and they're worth acknowledging. The benchmark data is drawn from published papers, which carry their own copyright restrictions. That limits how freely the full dataset can be shared, which in turn limits how widely other researchers can use it to train or test their own systems.
Alex: So even building the evaluation tool runs into the same intellectual property constraints that make scientific data sharing complicated generally.
Sam: It does. The authors flag it openly as a limitation rather than a solved problem. The broader vision—where AI systems can autonomously synthesize findings across thousands of papers, reading both text and figures—depends on resolving those access questions alongside the technical ones.
Alex: What would that actually look like in practice, if the technical problems were solved?
Sam: A researcher studying how a particular material behaves under different conditions could query a system that has read not just the text of thousands of papers, but the actual experimental data shown in their figures. The system could identify patterns across experiments that no individual researcher would have time to spot manually, and potentially suggest new experimental directions based on that synthesis.
Alex: That's a meaningful shift—from AI as a search tool to AI as something closer to a research collaborator.
Sam: That's the direction the paper points toward. But the authors are careful to frame it as a long-term trajectory, not an imminent capability. The benchmark itself is a step toward being able to measure progress rigorously—which is a necessary precondition for making that progress at all.
Alex: You have to be able to measure the gap before you can close it. Thanks for walking through this one, Sam.
Sam: Thanks for having me. If you've ever squinted at a figure in a paper and wondered whether a computer could make sense of it—the honest answer right now is: not reliably. But at least now there's a principled way to find out how close we're getting.
Alex: Thanks for listening to ResearchPod.