Efficient Multimodal Clinical Question Answering for Pulmonary Embolism Risk Assessment

This study investigates the utility of compact multimodal large language models (MLLMs) for assessing pulmonary embolism (PE), a critical condition requiring both rapid diagnosis and long-term risk stratification. The researchers utilize the INSPECT benchmark, a large-scale dataset containing over 23,000 CTPA studies linked to longitudinal electronic health records (EHRs). By reformulating eight diagnostic and prognostic tasks—such as PE presence, in-hospital mortality, and readmission risk—into structured clinical question-answering problems, the authors evaluate how well efficient MLLMs can reason over patient data.

The authors evaluate four compact MLLMs (Qwen3.5 4B, Gemma4 E4B, Gemma4 E2B, and MedGemma) across three input settings: CTPA-Only, EHR-Only, and CTPA+EHR. To handle volumetric CTPA data, the researchers convert 3D scans into 2D montages. The models are tested using both zero-shot and four-shot prompting, with performance measured by AUROC, AUPRC, accuracy, and F1. This framework emphasizes clinical reasoning by requiring the models to provide a binary answer and a confidence score for specific clinical queries.

The results indicate that EHR evidence is essential for effective PE risk assessment. Models consistently underperformed when relying solely on CTPA imaging, likely due to the complexity of interpreting volumetric data through 2D montages. Combining CTPA and EHR data yielded the highest performance, with Gemma4 E4B achieving the best results. The study also highlights a performance gap between diagnostic tasks, which are relatively well-modeled, and prognostic tasks, which require deeper longitudinal context. Additionally, the researchers identified significant challenges with class imbalance, where models often defaulted to majority-class predictions.

As healthcare systems seek to deploy AI in resource-constrained environments, compact MLLMs offer a privacy-preserving alternative to large, cloud-based models. This work demonstrates that these smaller models can be effectively tuned for clinical question-answering, provided they are fed the right combination of imaging and structured clinical history. The findings underscore the importance of multimodal integration in clinical decision support and provide a roadmap for evaluating AI models on complex, imbalanced medical datasets.