Omni-Perception Policy Optimization for Multimodal Emotion Reasoning

Multimodal Large Language Models (Omni-MLLMs) are increasingly used for Multimodal Emotion Reasoning (MER), where they must generate emotion predictions grounded in visual and acoustic signals. However, existing models often suffer from two critical failures: underutilization of multimodal cues and unfaithfulness, where models hallucinate modality-specific details (e.g., describing a facial expression) based on cues from a different modality (e.g., audio tone) rather than the actual visual input.

The authors propose Omni-Perception Policy Optimization (OPPO), a reinforcement learning framework designed to enforce grounded reasoning. OPPO introduces two primary mechanisms:

1. Omni-Perception Reward: This component decomposes ground-truth reasoning into fine-grained visual, acoustic, and emotion cues. It uses an Evidence-Routing Matrix to calculate the semantic similarity between these cues and the model's generated reasoning clauses, rewarding the model for explicitly covering human-annotated evidence.
2. Omni-Perception Loss: To combat hallucination, this loss compares the model's output distribution under full inputs versus unimodally masked inputs. It applies a KL-divergence penalty specifically to tokens identified as modality-specific evidence, forcing the model to change its output only when the corresponding modality is removed.

To evaluate these capabilities, the authors introduce MEP-Bench, a diagnostic benchmark that quantifies both the recall of multimodal cues (utilization) and the model's ability to maintain faithfulness under unimodal masking.

Experiments demonstrate that OPPO achieves state-of-the-art performance on standard benchmarks like MER-UniBench and MME-Emotion. More importantly, diagnostic results on MEP-Bench show that OPPO significantly outperforms baseline models in both utilization (higher recall of multimodal cues) and faithfulness (higher accuracy when probed with masked inputs). This confirms that explicitly optimizing for perception-grounded reasoning is essential for reliable multimodal emotion analysis.