Benchmark Evaluation of Feredated Learning on Multi-organ Images

Research Context and Motivation

Artificial intelligence has shown significant promise in medical image analysis, yet its translation into clinical practice remains stalled. A primary barrier is the nature of medical data: it is highly heterogeneous across institutions, sensitive, and subject to strict privacy regulations like HIPAA and GDPR. Federated Learning (FL) offers a solution by enabling collaborative model training without moving raw data between centers. However, the rapid proliferation of FL algorithms, combined with the lack of standardized, multi-organ, and multi-modal benchmarks, makes it difficult for researchers to objectively assess which algorithms perform best in real-world clinical environments.

The MobenFL Benchmark

To address these challenges, the authors developed MobenFL, a unified evaluation framework designed specifically for medical imaging. Unlike previous benchmarks that often focus on single organs or modalities, MobenFL provides a broad foundation for testing. It includes:

• Algorithmic Breadth: 20 cutting-edge FL algorithms, categorized into four types: those using additional network structures, regularization-driven methods, advanced aggregation strategies, and split learning architectures.
• Data Diversity: 22 medical imaging datasets covering 12 critical organs, including brain, lung, breast, and abdominal imaging, across various modalities like MRI, X-ray, and pathology.
• Multi-Dimensional Metrics: A systematic evaluation framework that measures performance (accuracy), algorithmic efficiency (communication and computation costs), and privacy protection capabilities.

Clinical Significance

By providing a standardized environment, MobenFL allows researchers to evaluate how different FL approaches handle the complexities of real-world clinical data, such as variations in imaging devices, disease types, and data distributions. The framework is designed to be extensible, with standardized input interfaces that allow for the direct integration of new algorithms and datasets. By open-sourcing this benchmark, the authors aim to accelerate the development of reliable, privacy-preserving AI tools that can be safely deployed across multiple healthcare institutions.