HSEmotion Team at the 11th ABAW Challenge: Multi-Task Learning and Ambivalence/Hesitancy Video Recognition

The 11th Affective Behavior Analysis in-the-Wild (ABAW) challenge focuses on two primary tasks: multi-task learning (MTL) for facial affect (valence, arousal, expressions, and action units) and video-level ambivalence/hesitancy (A/H) recognition. The authors investigate whether lightweight, frozen neural network backbones can compete with heavy, end-to-end fine-tuned models by shifting the focus from representation learning to systematic prediction calibration and multimodal fusion.

For the MTL task, the team utilizes two frozen lightweight backbones (MT-EmotiDDAMFN and MT-EmotiEffNet-B0). They implement a multi-stage post-processing pipeline that includes temporal Gaussian smoothing, per-class expression bias adjustment, and per-AU threshold tuning. These outputs are then fused using task-specific weights. For the A/H recognition task, the authors extend this approach by integrating facial features with HuBERT-large audio embeddings and RoBERTa-based text embeddings. These multimodal features are processed through frame-level MLPs, aggregated at the video level, and filtered using a global-text gate to produce a final binary classification.

The proposed ensemble significantly outperforms the official ConvNeXt baseline provided by the competition organizers. By avoiding the fine-tuning of large backbones, the authors achieve competitive results that rival heavier, end-to-end architectures. The study highlights that careful calibration—specifically backbone blending and temporal smoothing—provides greater performance improvements than simply increasing model complexity. The framework demonstrates high efficiency and deployment flexibility, making it suitable for real-world applications where computational resources are limited.