CADRE: Stable, Parameter-Efficient Adaptation of Medical Vision–Language Models with Bounded Forgetting and Prior Drift

Medical vision-language models (VLMs) like BiomedCLIP are powerful, but adapting them to new clinical imaging modalities often leads to catastrophic forgetting—where the model loses competence in previously learned tasks—and prior drift, where the model moves away from its reliable pretrained state. The authors investigate how to perform parameter-efficient continual adaptation while maintaining clinical stability, prioritizing the preservation of existing knowledge over raw leaderboard accuracy.

The authors introduce CADRE (Clinician-anchored, Domain-Robust, Efficient adaptation), a framework that keeps the backbone frozen and trains only a small fraction (0.23%) of parameters using LoRA. CADRE employs three primary stability mechanisms:

1. An online, self-scaling, similarity-aware elastic weight consolidation (EWC) term that bounds competence loss without the order-fragility typical of standard EWC.
2. An anchor-to-prior penalty that constrains the adapted model's embeddings to remain close to the frozen pretrained prior.
3. Calibration-preserving techniques, including label smoothing and evaluation-time weight averaging.

The authors validate this approach using a stress test involving three distinct breast cancer imaging modalities (histopathology, ultrasound, and chest radiography) under a multi-seed, multi-order protocol.

CADRE significantly outperforms baseline methods in stability metrics. It reduces catastrophic forgetting roughly sevenfold compared to the strongest regularized baseline and is the only method tested that achieves positive backward transfer. The framework demonstrates robust performance across different adaptation orders, supported by theoretical guarantees regarding the boundedness of consolidation mass and scale-invariance. By framing these results as monitorable stability properties, the authors provide a framework that aligns better with clinical-safety requirements than standard fine-tuning approaches.