The materials science literature encodes decades of experimental knowledge in figures, yet this visual record remains locked away and inaccessible to AI at scale. The core difficulty is structural: most scientific figures are compound, with a single caption describing multiple sub-panels simultaneously, making direct image-text pairing unreliable. We present MatMMExtract, an end-to-end open-source pipeline that resolves this by decomposing compound figures into individual sub-panels and generating structured, grounded annotations using a large language model guided by a curated materials science taxonomy. Applied to 14,810 open-access articles, MatMMExtract produces MatSciFig; 391,606 panel-level image-text pairs from 180,571 figures, each annotated with a sub-caption, a two-level visualisation category spanning 19 classes and over 100 subtypes, and a scientific summary. To enable accurate panel localisation, we introduce MaterialScope, a domain-specific detection dataset of 2,811 manually annotated materials science figures, on which a fine-tuned YOLO12-m detector achieves mAP_50 of 0.9227. Among six benchmarked language models, Gemini 3.1 Flash Lite delivers the best cost-quality trade-off for annotation generation, with 82% of outputs rated good and a hallucination rate of 4.8%. A dual-encoder retrieval baseline on MatSciFig achieves a 4.4 times improvement in R@1 over zero-shot CLIP, demonstrating the dataset's immediate utility for vision-language learning. All resources are released openly to the community.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new study on a surprisingly stubborn problem: how do you make the vast amount of visual data buried inside scientific papers actually usable by artificial intelligence?
Sam: So the core issue is that AI can't really "see" the experimental results hidden in those figures?
Alex: That's right, and the reason is more specific than you might expect. Think about how a typical science paper is laid out. You'll often see one big image that contains several smaller pictures inside it—labeled a, b, c, and so on—all described by a single block of text underneath. Researchers call these "compound figures."
Sam: And that's a problem because the AI can't tell which part of the description goes with which smaller image?
Alex: Exactly. The AI receives one large, jumbled picture and one block of text, with no clear map connecting them. So all that carefully recorded experimental data—microscope images, crystal structure diagrams, performance graphs—ends up locked away. The authors describe these as "visual fossils." The information exists, but it's effectively inaccessible.
Sam: So what did the researchers actually build to fix this?
Alex: They built an automated pipeline called MatMMExtract. The basic idea is to take those messy compound figures and split them into their individual panels, then attach a clear, specific description to each one. Think of it like a librarian who, instead of just filing a book by its title, opens every book, carefully cuts out each diagram, and creates a separate catalog entry for it—so you can search for a specific type of graph or microscope image directly.
Sam: That's a satisfying image. And the scale of what they processed is significant?
Alex: It is. They ran this pipeline over a large corpus of materials science papers and transformed more than 180,000 compound figures into nearly 400,000 individual, clean image-and-text pairs that an AI can actually learn from.
Sam: Why couldn't they just use existing AI tools for this? We already have systems that can spot objects in images—self-driving cars identify pedestrians, that sort of thing.
Alex: That's a fair question, and the answer comes down to what researchers call "domain shift." Imagine training someone to identify cars on a highway, then asking them to identify specific types of cells under a microscope. The skill of spotting objects doesn't automatically transfer, because the visual patterns are completely different. A general-purpose detector trained on everyday photographs simply doesn't know what a scanning electron microscope image looks like, or how a phase diagram is structured.
Sam: So they had to essentially teach the AI what materials science looks like from scratch?
Alex: Precisely. They assembled a custom dataset of around 2,800 figures, each one carefully labeled by human experts. To make sure those labels were reliable—that different experts looking at the same image would agree—they measured consistency using a standard statistical tool. Their agreement score was very high, close to perfect, which gave them confidence that the training data was solid.
Sam: And then they used that dataset to train a detection model?
Alex: Right. They tested several different model architectures—you can think of these as different "designs" for the AI's visual brain—and evaluated them on the trade-off between accuracy and speed. The one that performed best overall was compact enough to run on a single standard graphics card, which matters practically. A system that requires a supercomputer to process one paper isn't useful at scale.
Sam: So once the system has located and separated the individual panels, what happens next? How does it figure out what each panel actually shows?
Alex: This is where the text comes back in. The system uses a large language model—the kind of AI that's good at reading and summarizing text—to look at the paper's caption and the surrounding sentences where the author discusses that figure. From that context, it generates a focused description for each individual panel.
Sam: So it's not guessing. It's anchoring its description in what the human author actually wrote?
Alex: Exactly, and that's also the key limitation. The system is a sophisticated translator of existing knowledge. If the original caption is vague—something like "Figure 1: sample analysis"—then the generated description will be thin, because there's nothing richer to draw from. It cannot invent facts that aren't there.
Sam: That's an honest constraint. It's a tool for organization, not for filling in missing science.
Alex: That's the right way to frame it. And the researchers were clear about this. They also evaluated which language model produced the most reliable summaries—the one they settled on offered a good balance of accuracy and low rates of hallucination, which is the technical term for when an AI confidently states something that isn't true.
Sam: So where does this leave the field? If you now have 400,000 clean, labeled image-text pairs, what can you actually do with them?
Alex: The immediate application is better search. Right now, if a materials scientist wants to find all experiments involving a specific type of microstructure at low temperatures, they're largely searching through text. With a structured visual database, you could search by what things look like, not just by the words used to describe them.
Sam: And longer term?
Alex: The more ambitious goal the paper points toward is what researchers call inverse design. Instead of searching for a paper, you'd search for a recipe. You'd ask the database: across decades of published experiments, which combinations of materials and processing steps produced the best results for a specific property? The system could potentially surface patterns that no individual researcher would have the time to notice manually.
Sam: So we're moving from reading the literature to actually mining it for visual patterns?
Alex: That's a good way to put it. A lot of scientific progress has been recorded in figures that were never truly searchable. This work is an attempt to change that—to turn a passive archive into something you can actively query. Thanks for listening to ResearchPod.