Binesh Sadanadan, Vahid Behzadan
9 min
Abstract
Medical Vision Language Models VLMs suffer from two failure modes that threaten safe deployment mis calibrated confidence and sensitivity to question rephrasing. We show they share a common cause, proximity to the decision boundary, by benchmarking five uncertainty quantification methods on MedGemma 4BIT across in distribution MIMIC CXR and outof distribution PadChest chest X ray datasets, with cross architecture validation on LLaVA RAD7B. For well calibrated single model methods, predictive entropy from one forward pass predicts which samples will flip under rephrasing AUROC 0.711 on MedGemma, 0.878 on LLaVARAD p 10 4, enabling a single entropy threshold to flag both unreliable and rephrase sensitive predictions. A five member LoRA ensemble fails under the MIMIC PadChest shift 42.9 ECE, 34.1 accuracy, though LLaVA RAD s ensemble does not collapse 69.1. MC Dropout achieves the best calibration ECE 4.3 and selective prediction coverage 21.5 at 5 risk, yet total entropy from a single forward pass outperforms the ensemble for both error detection AUROC 0.743 vs 0.657 and paraphrase screening. Simple methods win.
Alex: Huh, that's solid. But what about grouping several tweaked versions together, like a team vote for better decisions?
Sam: They tried that too—a deep ensemble, where five separately trained LoRA versions average their predictions. In-distribution, on matching data, it did well. But out-of-distribution, four of the five performed near random, around chance level, while one generalized decently. Averaging let the weak ones drag down the strong one, worsening calibration to 43%—worse than the base.
Alex: So the team idea backfires when the members don't all adapt to new data. Makes sense why singles might be safer.
Sam: Exactly. Single-model methods like the targeted LoRA stayed stable even under image corruptions—things like added noise, blur, or brightness changes that mimic real clinical variations. For most corruptions, miscalibration hovered low, between 4% and 7%, close to clean performance. This robustness comes from the model's image encoder being pre-trained on diverse web photos.
Alex: So a single quick check flags both issues, and the tweaks hold up under real-world image glitches. But how exactly did they measure if this uncertainty score was good at spotting errors versus other ways?
Sam: They tested several approaches to gauge how unsure a model is. One common way is to bucket predictions by confidence level, then check if higher-confidence groups are actually more accurate on average—if not, the model is miscalibrated. The gap between stated confidence and real accuracy, averaged across those buckets, is called Expected Calibration Error, or ECE; lower is better.
Alex: Got it—like sorting test scores into groups and seeing if kids who say they're super sure really get them right more often.
Sam: Right. For spotting outright wrong answers, they ranked predictions by uncertainty and saw how well high-uncertainty ones matched errors. The paper's entropy method hit about 0.74 on that scale, working roughly 15% better than alternatives like team-voting ensembles. Same uncertainty score flagged paraphrase flips too, at around 0.71—still a clear edge over others. Both stem from predictions teetering near the yes-no divide, where even slight wording shifts tip the balance.
Alex: Huh, so one metric catches two linked flaws. What about breaking down the uncertainty itself into types?
Sam: The total uncertainty splits into two parts: one from inherent fuzziness in the data, like blurry images that no model could nail perfectly, and another from the model's own gaps in knowledge. Entropy captures the first reliably, while tricks like dropout runs estimate the second by simulating varied model views. This decomposition helps prioritize fixes—train more on noisy data or fill knowledge holes.
Alex: So not all uncertainty is equal; some you live with, some you engineer away. Does this hold across different hospitals' scans?
Sam: Yes, cross-site tests on MIMIC-CXR to PadChest showed single-model entropy staying reliable, unlike ensembles where mismatched team members tanked performance. They checked each LoRA adapter separately on the new PadChest scans. Four out of five performed poorly, near random guessing, because each was tuned too tightly to the original MIMIC training images. The fifth generalized decently, but when averaged, the weak ones pulled the prediction toward confusion.
Alex: Right, like a sports team where one star carries it, but the bench weighs everything down.
Sam: Exactly. They also tried other grouping tricks, like averaging raw scores before softening or simple majority vote, but none fixed it—the poor performers dominated. MC Dropout keeps a small random fraction of connections turned off during prediction runs—like flipping a few light switches unpredictably each time to see varied model opinions. They average ten such runs from one trained adapter. It slightly edges softmax entropy on some image glitches, holding miscalibration to 4-5% where others rise, but adds compute time.
Alex: So multiple quick reruns give a steadier view without full retraining.
Sam: Yes. Temperature scaling is simpler: it learns one number to dial back overconfident scores, like adjusting a thermostat so hot reads cool off and cold ones warm up. Trained on a bit of held-out data, it helps in-distribution but doesn't transfer well across sites—calibration from one hospital's scans worsens on another's. Single softmax entropy stayed low, 4-7% across most corruptions and strengths. Ensembles bombed everywhere at 40%+, even clean. MC Dropout held close, a small gain on heavy noise.
Alex: So for clinics, stick to one reliable check over fancy groups. That feels practical.
Alex: That feels practical. But to wrap this up, what are the main limitations here—things that might not carry over to other situations?
Sam: A few key ones stand out. The tests focus on shifting from one chest X-ray dataset, MIMIC, to another, PadChest—real hospitals differ more widely. Questions are simplified to yes-or-no answers, skipping open-ended ones doctors often use. And while LoRA tweaks help, ensemble versions succeed on some models like LLaVA-RAD but flop on others like MedGemma, so results aren't universal.
Alex: Right, so binary questions and one main data shift. Does the paper offer any other safety checks beyond entropy?
Sam: Yes, they explore conformal prediction as a backup. It sets prediction ranges—like saying 'yes, no, or maybe'—with a math guarantee that it covers the right answer a set percentage of the time, no matter the data quirks. You calibrate it on a small held-out chunk, then it builds flexible sets around predictions. On corrupted images, the targeted LoRA version holds or beats those guarantees, unlike the base model which falls short.
Alex: Huh, guaranteed coverage even on noisy scans. And in terms of real-world payoff?
Sam: The evidence points to a straightforward clinic workflow: run a quick entropy check—if high, send to a radiologist for review. At 40% coverage, meaning you trust the four lowest-risk answers out of ten, error rates drop about 63% and flip risks by around 39%. This screens both mistakes and rephrase wobbles without fancy teams, using one model pass for speed. It holds across models too—entropy spots flips on LLaVA-RAD just as well.
Alex: So auto-flag the shaky ones, review the rest—cuts workload while boosting safety.
Sam: Exactly. The paper recommends flagging high-entropy cases for human review, boosting trust without complexity. It's a measured advance in making these AIs safer for clinics.
Alex: A measured path forward, grounded in these tests. Thanks, Sam—that's a clear picture of where medical image AI stands on reliability.
Sam: My pleasure, Alex. This work shows simple checks can build trust step by step. Thanks for listening to ResearchPod.