Hyunjae Kim, Dain Kim, Pan Xiao, Serina S. Applebaum, Younjoon Chung, Xuguang Ai, Yu Yin, Roy Jiang, Yuexi Du, Yawen Wei, Yiming Kong, Tuo Guo, Zhiyuan Cao, Mengmeng Du, Yuelei Fu, Yan Hu, Rui Shi, Gui Yang, Kevin W. Jin, Yuntian Liu, Yuxuan Tian, Jonathan Marquez, Zhen Chen, Sheng Zhang, Hoifung Poon, Hua Xu, Jaewoo Kang, Qingyu Chen
5 min
Medical foundation models are often limited by the scarcity of high-quality, large-scale, and publicly accessible multimodal data. While PubMed Central (PMC) provides a vast repository of biomedical literature, existing datasets derived from it often suffer from low medical relevance, poor image-text alignment, and static, non-reproducible curation methods. This paper asks whether a systematic, modular framework can transform this raw literature into high-fidelity training data that improves the performance of medical multimodal foundation models.
The authors developed MedPMC, a five-stage automated pipeline designed to process 6.1 million PMC articles. The stages include: (1) initial screening to filter for medical content, (2) multi-panel figure detection, (3) figure separation to isolate individual subfigures, (4) caption separation and alignment using a multimodal large language model (MLLM), and (5) medical figure classification to ensure clinical relevance. The framework is modular, allowing for continuous updates as new literature is published and individual components are refined. The authors validated the pipeline using a combination of existing benchmarks, synthetic data, and manual expert review.
MedPMC successfully curated 11 million high-quality medical image-text pairs. Manual review confirmed that 95.3% of the curated images were medically relevant, a significant improvement over prior datasets. When used to train a CLIP-style vision-language model, MedPMC-trained models achieved state-of-the-art results across 26 medical benchmarks, improving average zero-shot AUC by 7.1 percentage points compared to existing baselines. Furthermore, the model demonstrated superior performance in clinical tasks, such as morphology-to-image retrieval using real-world dermatology photographs from the Yale New Haven Health System, and improved medical visual question-answering when used as an encoder in an MLLM.
MedPMC provides a scalable, reproducible, and sustainable infrastructure for medical AI development. By moving away from static, one-time dataset construction, it enables the continuous integration of new biomedical knowledge into foundation models. The framework's ability to produce high-fidelity data that generalizes across 11 medical specialties suggests that literature-derived curation is a viable path toward bridging the gap between public benchmarks and real-world clinical utility.
Medicine is inherently multimodal, requiring clinicians to synthesize information across diverse data streams. Yet the development of multimodal foundation models is constrained by limited access to large-scale, high-quality clinical data. Although PubMed Central (PMC) offers a complementary source of expert-authored image-text data, existing PMC-derived resources remain limited in fidelity, reproducibility, and clinical validation. We introduce MedPMC, an automated, continuously updatable framework that transforms permissively licensed literature into high-fidelity infrastructure for medical multimodal models. Applied to 6.1 million PMC articles, MedPMC curated 11 million medical image-text pairs. Component evaluations showed strong performance for initial screening (F1 = 93.2), multi-panel figure detection (F1 = 96.5), figure separation (mAP = 89.8), caption separation and alignment (F1 = 81.4; ROUGE-L = 85.3), and medical figure classification (F1 = 96.5). Manual review by five annotators, three with medical training, found 95.3% of MedPMC images medically relevant, versus 19.7% in a prior PMC-derived dataset. Across 26 benchmarks spanning 11 specialties, a MedPMC-trained CLIP-style model improved average zero-shot AUC by 7.1 percentage points over the strongest architecture-matched biomedical CLIP baseline despite using fewer than half as many image-text pairs. As the vision encoder in a multimodal large language model, it improved medical visual question-answering by 1.9 and 16.9 percentage points across two benchmarks. In 10,524 Yale New Haven Health System dermatology photographs, it improved morphology-to-image retrieval Recall@5 by 11.7 percentage points. These findings show that high-fidelity literature curation strengthens medical multimodal foundation models across benchmark and clinical settings. We publicly release the framework, corpus, benchmarks, and pretrained models.
Alex: Oh—so that's why the noisier datasets were struggling. They weren't breaking apart those complex figures, so the AI was essentially trying to learn from a jumbled mess where one caption might describe three different images at once.
Sam: Precisely. And once you fix that, the model's ability to do what researchers call "visual question-answering" improves markedly. That's where the AI looks at a medical image and answers a specific clinical question about it—the kind of task that would actually be useful to a doctor.
Alex: That makes sense as a benchmark. But how do we know this holds up with real patient data, not just carefully prepared test sets?
Sam: That's exactly the right question—it's what researchers sometimes call the benchmark-to-bedside gap. To address it, they tested the model using over ten thousand real dermatology images from clinical settings. The MedPMC-trained model was substantially better at finding the right images based on text descriptions than models trained on standard, unfiltered data. So it wasn't just learning textbook patterns—it was learning to recognize the kind of variation you actually see in real patients.
Alex: That distinction matters. A model that only knows what disease looks like in a curated textbook image might not recognize it when a photo is taken under different lighting, or at a different angle, or on a different skin tone.
Sam: Exactly. And that's what makes the curation approach significant beyond the technical results. By building a pipeline that prioritizes clinical relevance from the very start, the researchers are making a case that the quality of the training data shapes the reliability of the model in ways that simply adding more data cannot fix. It's a more disciplined foundation for medical AI going forward.
Alex: That's a useful reframe—less about how much data you have, and more about whether the data actually reflects the complexity of real medical practice. Thanks for walking us through it, Sam, and thanks to everyone for listening to ResearchPod.