FM2: Unified Federated Foundation Models for Heterogeneous Multimodal Medical Imaging

Building foundation models for medical imaging is hindered by privacy regulations that prevent centralized data pooling. Existing federated approaches often struggle with Imaging Modality Heterogeneity, where institutions operate with different imaging hardware (e.g., MRI vs. CT) and non-IID label distributions. The authors investigate how to build a unified, multi-task federated foundation model that can handle both overlapping and disjoint modality configurations without requiring raw data exchange.

The authors propose FM2, a framework that trains a core visual backbone from scratch to maintain medical domain fidelity. The architecture features two key innovations:

• Dual Mixture-of-Experts (MoE): A Class-wise MoE handles personalized category knowledge locally, while a Domain-wise MoE captures shared cross-modality representations that are globally aggregated.
• Heterogeneous Modality Alignment (HMA): A regularizer that constrains local domain experts toward a global consensus, ensuring stable convergence even when clients have fundamentally different imaging characteristics.
• Caption-Enhanced Learning (CEL): To address scenarios where clients have completely disjoint modalities, the model uses locally retained GPT-4o-generated captions as a textual semantic bridge. This allows the model to align visual features with shared diagnostic language, facilitating representation transfer across disparate clinical sites.

FM2 demonstrates consistent superiority over state-of-the-art federated baselines across classification, caption-supervised learning, and medical Visual Question Answering (VQA). The authors provide formal proofs showing that the framework achieves a convergence rate of O(1/√T) and explicit generalization bounds. Experimental results on the newly constructed MIMH benchmark confirm that the dual-MoE structure effectively disentangles class-level personalization from modality-level consensus, enabling strong out-of-modality generalization.