Evaluating medical AI under missing information: same-provider judges and human raters change apparent safety

As medical AI models reach high performance on standardized, closed-ended exams like MedQA, there is a growing concern that these benchmarks fail to measure 'clinical readiness.' This study investigates how frontier models behave when faced with open-ended clinical conversations where critical information is missing. Specifically, it asks: do models recognize when they lack sufficient information to provide a safe, definitive answer, or do they over-commit and hallucinate?

To test this, the author introduced a 'missing-information' stress test by truncating the final user turn in clinical conversations from HealthBench. A robust model should identify the missing data and ask for clarification rather than providing a confident diagnosis. Because open-ended responses cannot be graded with a simple answer key, the study evaluated model outputs using a four-provider LLM-judge panel and compared these results against a blinded, clinician-anchored reference. The study also performed a rigorous audit of the perturbations to distinguish between genuinely underdetermined clinical tasks and administrative or answerable prompts.

1. Evaluator Bias: The choice of judge significantly impacts the safety rating. LLM judges show a 'same-provider' preference, where they rate their own provider's models more favorably. Even after adjusting for general leniency, this bias is large enough to change the relative safety rankings of the models.
2. Permissiveness Gap: LLM judges are consistently more permissive than human clinicians. While models were credited with 'appropriate uncertainty' by LLM judges in 66–84% of cases, human clinicians were much stricter, crediting such behavior in only 52% of cases.
3. Calibration vs. Knowledge: A closed-ended MedQA anchor test confirmed that the models possess high medical knowledge. The safety failures observed in the open-ended probe are therefore attributed to poor calibration—the inability to recognize when to abstain—rather than a lack of medical facts.