General Incomplete Multimodal Learning via Dynamic Quality Perception

Multimodal learning models often assume that all input modalities are perfectly available and clean. In real-world applications, however, models frequently encounter two distinct problems: inter-modality missing (where an entire modality is absent) and intra-modality degradation (where a modality is present but corrupted by noise). Existing methods typically treat these as separate issues or use two-stage pipelines, which can lead to optimization conflicts and poor performance. This paper asks: can we develop a unified framework that handles both types of incompleteness simultaneously through dynamic quality perception?

The authors propose General Incomplete Multimodal Learning (GIML), a framework that models all forms of missingness as a continuous spectrum of information degradation. GIML introduces two key components:

1. Noise-Semantic Decoupled (NSD) Module: This module treats modality features as probabilistic distributions rather than deterministic vectors. By separating task-relevant semantic information (the mean) from degradation-induced uncertainty (the variance), the model learns representations that are invariant to noise, improving robustness to unseen corruption patterns.
2. Noise-aware Quality Estimator (NQE): This component learns to map corrupted features to specific noise intensities using controlled noise injection during training. By explicitly predicting the degradation level, the model can assign adaptive fusion weights, ensuring that heavily corrupted modalities contribute less to the final representation.

By unifying the treatment of missing and corrupted data, GIML avoids the sub-optimal performance of sequential pipelines. The ability to dynamically adjust modality weights based on estimated quality allows the model to remain effective even when data quality fluctuates significantly. This approach provides a more flexible and robust solution for real-world multimodal systems, such as autonomous driving or emotion recognition, where sensor failure or environmental noise is common.