Evaluating and Understanding Model Editing for Medical Vision Language Models

As medical vision-language models (VLMs) are increasingly deployed in clinical settings, they inevitably encounter errors due to evolving clinical protocols and patient data. While model editing offers a lightweight alternative to full retraining, its performance in high-stakes medical environments remains underexplored. This paper introduces M3Bench, a clinically grounded benchmark designed to evaluate model editing across 10 tasks covering reliability, locality, generality, and temporal consistency. The authors evaluate four representative editing methods—gradient-based (MEND, LoRA) and memory-based (GRACE, BalancEdit)—across six different VLM backbones to determine how well these methods handle real-world clinical challenges.

The study reveals that no single editing method consistently outperforms others across all clinical criteria. Gradient-based editors like LoRA achieve high reliability and strong knowledge transfer to similar cases, but they frequently exhibit poor locality, meaning they inadvertently corrupt unrelated clinical knowledge. Conversely, memory-based editors like BalancEdit excel at preserving locality but often fail to generalize to complex, compositional clinical scenarios. Furthermore, the authors demonstrate that these performance gaps are rooted in the latent space geometry of medical VLMs. Medical models tend to concentrate representations into a narrow, anisotropic cone, which makes localized editing intrinsically difficult. Gradient-based methods tend to induce global concept drift, while memory-based methods rely on binary activation regions that can miss interleaved clinical concepts.

This research provides a rigorous stress test for post-deployment model adaptation in healthcare. By taxonomizing editing methods and linking their failure modes to the underlying geometry of VLM latent spaces, the authors offer actionable guidance for developers. The findings suggest that there is no 'one-size-fits-all' solution for medical model editing, and that future efforts must focus on balancing the trade-off between precise, localized corrections and the ability to generalize across the diverse, compositional nature of clinical data.