Subham Ghosh, Shubham Tiwari, Mohammad Ibrahim, Abhishek Tewari
5 min
Abstract
The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale. The core difficulty is structural: most scientific figures are compound, with a single caption describing multiple sub-panels simultaneously, making direct image-text pairing unreliable. We present MatMMExtract, an end-to-end open-source pipeline that resolves this by decomposing compound figures into individual sub-panels and generating structured, grounded annotations using a large language model guided by a curated materials science taxonomy. Applied to 14,810 open-access articles, MatMMExtract produces MatSciFig; 391,606 panel-level image-text pairs from 180,571 figures, each annotated with a sub-caption, a two-level visualisation category spanning 19 classes and over 100 subtypes, and a scientific summary. To enable accurate panel localisation, we introduce MaterialScope, a domain-specific detection dataset of 2,811 manually annotated materials science figures, on which a fine-tuned YOLO12-m detector achieves mAP_50 of 0.9227. Among six benchmarked language models, Gemini 3.1 Flash Lite delivers the best cost-quality trade-off for annotation generation, with 82% of outputs rated good and a hallucination rate of 4.8%. A dual-encoder retrieval baseline on MatSciFig achieves a 4.4 times improvement in R@1 over zero-shot CLIP, demonstrating the dataset's immediate utility for vision-language learning. All resources are released openly to the community.
Sam: So they had to essentially teach the AI what materials science looks like from scratch?
Alex: Precisely. They assembled a custom dataset of around 2,800 figures, each one carefully labeled by human experts. To make sure those labels were reliable—that different experts looking at the same image would agree—they measured consistency using a standard statistical tool. Their agreement score was very high, close to perfect, which gave them confidence that the training data was solid.
Sam: And then they used that dataset to train a detection model?
Alex: Right. They tested several different model architectures—you can think of these as different "designs" for the AI's visual brain—and evaluated them on the trade-off between accuracy and speed. The one that performed best overall was compact enough to run on a single standard graphics card, which matters practically. A system that requires a supercomputer to process one paper isn't useful at scale.
Sam: So once the system has located and separated the individual panels, what happens next? How does it figure out what each panel actually shows?
Alex: This is where the text comes back in. The system uses a large language model—the kind of AI that's good at reading and summarizing text—to look at the paper's caption and the surrounding sentences where the author discusses that figure. From that context, it generates a focused description for each individual panel.
Sam: So it's not guessing. It's anchoring its description in what the human author actually wrote?
Alex: Exactly, and that's also the key limitation. The system is a sophisticated translator of existing knowledge. If the original caption is vague—something like "Figure 1: sample analysis"—then the generated description will be thin, because there's nothing richer to draw from. It cannot invent facts that aren't there.
Sam: That's an honest constraint. It's a tool for organization, not for filling in missing science.
Alex: That's the right way to frame it. And the researchers were clear about this. They also evaluated which language model produced the most reliable summaries—the one they settled on offered a good balance of accuracy and low rates of hallucination, which is the technical term for when an AI confidently states something that isn't true.
Sam: So where does this leave the field? If you now have 400,000 clean, labeled image-text pairs, what can you actually do with them?
Alex: The immediate application is better search. Right now, if a materials scientist wants to find all experiments involving a specific type of microstructure at low temperatures, they're largely searching through text. With a structured visual database, you could search by what things look like, not just by the words used to describe them.
Sam: And longer term?
Alex: The more ambitious goal the paper points toward is what researchers call inverse design. Instead of searching for a paper, you'd search for a recipe. You'd ask the database: across decades of published experiments, which combinations of materials and processing steps produced the best results for a specific property? The system could potentially surface patterns that no individual researcher would have the time to notice manually.
Sam: So we're moving from reading the literature to actually mining it for visual patterns?
Alex: That's a good way to put it. A lot of scientific progress has been recorded in figures that were never truly searchable. This work is an attempt to change that—to turn a passive archive into something you can actively query. Thanks for listening to ResearchPod.