Ayush Debnath, Ruelia Saha, Sudip Misra
4 min
OmniMed-FL addresses the challenge of performing clinical condition classification across decentralized hospital networks while adhering to strict data privacy regulations like HIPAA and GDPR. While standard machine learning models often rely on centralized data, federated learning (FL) allows institutions to train models collaboratively without sharing sensitive patient records. OmniMed-FL extends this by enabling multimodal fusion—combining visual data (chest radiographs) with textual data (clinical notes)—within a federated framework.
The researchers developed a controlled proxy corpus consisting of 3,000 chest radiographs paired with class-conditioned synthetic clinical notes. The framework evaluates eight different fusion strategies, three initialization methods, and four missing-text imputation rules across 3 to 20 hospital clients. To handle the challenges of federated learning, the authors implemented anti-collapse components, including a class-balanced sampler and an entropy-diversity penalty, to ensure the model maintains predictive diversity across the five clinical classes (Normal, Pneumonia, COVID-19, Pleural Effusion, and Cardiomegaly).
The study finds that multimodal fusion significantly outperforms unimodal approaches, achieving higher macro-F1 scores on both synthetic and radiograph-based corpora. The researchers observed that label skew (non-IID data distribution) is a more significant driver of performance degradation than the number of participating clients. While federated training incurs additional communication and computational costs—specifically 2.3 times the model state of text-only models—it successfully retains high performance compared to centralized baselines. The results suggest that warm-start initializations and robust regularization are critical for maintaining performance in decentralized, multimodal settings.
This work provides a practical blueprint for hospitals to collaborate on diagnostic models without exposing raw patient data. By quantifying the trade-offs between communication overhead, computational cost, and model accuracy, OmniMed-FL offers a foundation for future telehealth and diagnostic workflows that require the integration of diverse clinical data types.
Simultaneous assessment of medical imaging and patient records is often required in clinical diagnosis. However, standard machine learning algorithms cannot analyze these data types together. Meanwhile, compliance with HIPAA and GDPR can constrain centralized aggregation of sensitive patient data. This leaves a crucial void of secure fusion of visual and textual context across distant networks. Thus, we present OmniMed-FL, a controlled systems study of multimodal federated learning for five-class clinical condition classification (Normal, Pneumonia, COVID-19, Pleural Effusion, Cardiomegaly). Our proxy corpus pairs 3,000 public chest radiographs with 3,000 class-conditioned synthetic notes, matched by class, not by patient. The framework benchmarks eight fusion strategies, three initializations, four missing-text imputation rules, and matched federated baselines under non-IID Dirichlet partitioning across 3 to 20 hospital clients. As all notes are synthetic and pairing is not patient-level, these are descriptive proxy comparisons, not estimates of diagnostic performance or deployment readiness. Within those limits with clients ($K=5$) and severe skew ($α=0.1$), local-only training achieves a macro-F1 score of 0.297, FedAvg achieves $0.662\pm0.074$, FedProx $0.737\pm0.085$, a matched FedMME-style one-shot ensemble $0.647\pm0.080$, and our SCAFFOLD-AdamW adaptation $0.070\pm0.015$, the 0.075 FedProx-FedAvg gap falling inside the wider of the two two-seed standard deviations. Over a $4\times3$ grid, label skew costs up to 0.27 F1 whereas a near-sevenfold client increase costs at most 0.10, while bidirectional volume grows linearly to 183.5 GiB at $K=20$. Multimodal fusion leads on both corpora, scoring 0.956 against 0.934 for text and 0.664 for images on the synthetic corpus and 0.906 against 0.880 and 0.737 on the radiograph corpus, for $2.3\times$ the model state of text alone.
Sam: [nodding] Exactly. And part of that ablation covers their anti-collapse regularization stack — a class-balanced sampler combined with an entropy-diversity penalty designed to stop the model defaulting to the majority class under severe skew.
Alex: [analytical] Did that regularization actually move the needle?
Sam: [measured] It depends on the regime. Under severe label imbalance, it was essential for maintaining class diversity in the predictions. Under moderate skew, those components were largely inert. Think of it as a guardrail you only need when the distribution is heavily biased — it does not add much when conditions are already reasonable. [[RP_SECTION:communication-overhead-costs|Communication Overhead Costs]]
Alex: [shifting focus] What about communication overhead? Moving multimodal models across a network at scale has to be expensive.
Sam: [precise] It is. Bidirectional traffic grows linearly with client count — at 20 clients they exceed 180 gigabytes. Multimodal fusion carries more than double the model state size of a text-only branch. That is the concrete cost you accept for the performance gain, and it is the kind of number a system designer needs to plan around before scaling up.
Alex: [reflective] So the central tension is communication efficiency versus the diagnostic lift you get from combining both modalities.
Sam: [affirming] That is exactly it. And the study makes that trade-off legible in a way it wasn't before. The authors are explicit that the next steps require formal differential privacy guarantees and asynchronous aggregation — and that moving from this controlled benchmark to production will mean handling the actual missingness and discordance of real patient-paired records, which is a substantially harder problem.
Alex: [deliberate] So this is a foundational step in quantifying the trade-offs, not a claim of clinical readiness.
Sam: [quiet confidence] Precisely. It establishes how these systems behave under distributed constraints — which is exactly what the field needs before tackling the messier, higher-stakes conditions of real clinical deployment. A meaningful, incremental step in a genuinely difficult problem space.