Evaluating Multi-Turn Multimodal Diagnostic Reasoning on Challenging Real-World Clinical Cases

Clinical diagnosis is a dynamic, iterative process where physicians integrate multimodal data—such as patient history, lab results, and medical imaging—over time. While multimodal large language models (MLLMs) have shown promise in isolated medical tasks, their ability to perform sustained, multi-turn diagnostic reasoning remains poorly understood. This paper introduces ClinMM-Bench, a new benchmark designed to evaluate how well MLLMs handle the progressive disclosure of clinical information and the subsequent refinement of diagnostic hypotheses.

ClinMM-Bench consists of 1,089 challenging, real-world clinical cases across eight specialties, incorporating 3,760 medical images. The researchers developed a two-level evaluation framework:

1. Diagnostic Accuracy: A dual-LLM consensus mechanism scores models on their final diagnostic output (0 to 2 scale).
2. Reasoning Quality: The study decomposes model-generated reasoning into atomic facts, measuring fact recall, hallucination rates, and fact density to assess the reliability and completeness of the underlying clinical logic.

Fifteen representative models, including proprietary, open-weight, medical-specific, and reasoning-focused variants, were tested to identify performance gaps and common failure modes.

Proprietary models generally outperformed open-weight models, yet even the top-performing models showed limited rates of completely correct diagnoses. The analysis revealed that while models can often identify plausible diagnostic directions, they frequently fail to synthesize evolving information correctly. The researchers identified five primary failure modes: information synthesis failure, knowledge mapping errors, perception errors, premature closure (sticking to an early, incorrect hypothesis), and visual hallucinations. Furthermore, the study found that medical-domain adaptation and explicit reasoning-based training did not consistently improve performance, suggesting that simply scaling or fine-tuning models is insufficient to overcome these complex reasoning challenges.