When Does Quality-Aware Multimodal Fusion Matter? A Leakage-Safe Diagnostic for Decision-Level Dependence

Many multimodal systems incorporate 'quality-aware' fusion, where reliability scores (e.g., signal-to-noise ratios) are used to weight different modalities. However, it is often unclear if these models actually use these scores to make better decisions at inference time, or if the performance gains are merely due to correlations in the training data. This paper introduces a diagnostic to test whether a model's predictions are truly dependent on the alignment between quality signals and specific input instances.

The authors propose a 'leakage-safe' diagnostic: after training a multimodal model, they freeze the model parameters and the input evidence. They then shuffle (permute) the reliability scores across the test set. If the model relies on these scores to route information, the performance should drop significantly when the alignment between the quality signal and the input is broken. They evaluate this on the StressID and CMU-MOSEI datasets, comparing the performance of 'Clean-Q' (original alignment) against 'Broken-Q' (shuffled alignment).

Across both datasets, the authors found that shuffling native quality signals resulted in negligible performance changes. This indicates that the models were not meaningfully using the quality estimates to guide their decisions, even though 'oracle' analysis showed that better per-example routing was theoretically possible. Crucially, when the authors used 'positive controls'—where quality signals were synthetically constructed to perfectly track unimodal correctness—the models showed significant performance drops when the signals were shuffled. This proves that the fusion architectures are capable of using quality information, but the native quality metrics currently used in the field fail to provide the necessary predictive power.