Beyond Modality Fusion: Deep Ensembles for Multimodal Classification

This paper investigates the effectiveness of heterogeneous deep ensembles (HeDEs) for multimodal classification, challenging the standard paradigm of explicit modality fusion. Unlike late-fusion approaches that concatenate features from different modalities, the authors train unimodal networks independently and aggregate their output logits. To systematically evaluate this, the authors introduce a synthetic multimodal framework that allows for precise control over the number of modalities and their predictive strength, enabling the study of optimization challenges like modality imbalance.

The authors demonstrate that deep ensembles consistently outperform state-of-the-art late-fusion techniques, including those specifically designed to mitigate modality imbalance (e.g., gradient modulation and loss regularization). This performance advantage is robust across both synthetic and real-world datasets, such as CREMA-D and Kinetics-400. A key contribution is a heuristic for selecting the number of models per modality in an ensemble, which avoids computationally expensive exhaustive searches. The study reveals a distinct scaling pattern: under extreme modality imbalance, homogeneous ensembles of the stronger modality are initially superior, but as the ensemble size grows, incorporating models from the weaker modality becomes beneficial. The authors also establish scaling laws that allow researchers to predict the asymptotic performance of these ensembles based solely on the validation losses of the individual unimodal models.

Multimodal learning is often hindered by the 'greedy' nature of neural networks, where one modality dominates the learning process, leading to inferior performance compared to unimodal baselines. By showing that simple, independent training and logit-averaging can surpass complex fusion architectures, this work provides a powerful, scalable, and easier-to-optimize alternative for multimodal tasks. The proposed heuristic and scaling laws offer practical guidance for practitioners to design efficient ensembles without the need for extensive trial-and-error.