Modality-Aware Out-of-Distribution Detection for Multi-Modal Action Recognition

Multi-modal action recognition models often outperform uni-modal counterparts by leveraging complementary information from sources like audio and optical flow. However, these systems frequently struggle to identify out-of-distribution (OOD) inputs, often defaulting to off-the-shelf uni-modal OOD detectors that fail to exploit the rich, inter-modal relationships available at inference time. This paper investigates how to better utilize these multi-modal signals to improve OOD detection robustness.

The authors observe that for in-distribution (ID) samples, there is a strong, predictable linear relationship between the predictions of individual uni-modal heads and the final multi-modal head. Conversely, this relationship breaks down when the model encounters OOD samples. The researchers propose a hybrid, post-hoc detector that combines three signals:

1. A modality-aware score based on the error of this linear mapping.
2. A feature-space score that identifies off-manifold samples using Eigen decomposition of the concatenated feature covariance matrix.
3. The model's own output logits.

By normalizing these scores and integrating them, the detector creates a unified OOD score without requiring changes to the training process.

Findings

Evaluated on the Multi-OOD benchmark, the proposed method consistently outperforms existing state-of-the-art approaches across various datasets and modalities. For example, on Kinetics-600 Far-OOD experiments, the method achieved a 13.5% improvement in False Positive Rate (FPR) compared to previous baselines. The results demonstrate that explicitly modeling the interaction between modalities at inference time provides a critical cue for detecting anomalies that uni-modal detectors miss.