MEDBENCH V5: A DYNAMIC, PROCESS-ORIENTED, AND HALLUCINATION-AWARE BENCHMARK FOR CLINICAL MULTIMODAL MODELS

Traditional medical AI benchmarks rely on static, single-turn question-answering tasks that fail to capture the iterative, uncertain, and evidence-based nature of real-world clinical practice. This paper asks: how can we move beyond outcome-based evaluation to assess the process-level reasoning, robustness, and hallucination trajectories of clinical multimodal models in dynamic environments?

The authors introduce MedBench v5, a comprehensive evaluation framework that shifts from static QA to a dynamic, process-oriented protocol. The framework is built on two pillars:

1. Dual-Dimensional Framework: It evaluates models across 14 Clinical Cognitive Responsiveness (CCR) dimensions (e.g., clinical reasoning, multimodal perception, multi-agent collaboration) and 4 Medical Atomic Skills (MAS) environments (DataAgent, RAGAgent, DeepResearch, and SafetyAgent).
2. Stress-Audit-Tracing Protocol: The benchmark introduces three "information-flow stressors"—information omission, contradiction injection, and evidence delay—to perturb clinical scenarios. It then uses a five-node audit protocol to track how models handle these stressors across reasoning steps, specifically monitoring how hallucinations initiate, propagate, and anchor in the final diagnostic output.

Experiments on frontier models reveal a significant "knowledge-practice gap." While models may achieve high accuracy on standard tasks, they often exhibit fragile reasoning when faced with dynamic clinical stressors. Specifically, models struggle with contradiction detection, updating diagnoses in light of new evidence, and preventing the propagation of hallucinations. Interestingly, the study finds that final evidence grounding can appear stable even when the underlying reasoning process has failed, suggesting that models may arrive at correct conclusions through flawed or unsupported reasoning paths.