Audio-Text Cross-Attention with Psycholinguistic Support Features for Ambivalence/Hesitancy Recognition

How can machine learning systems effectively identify ambivalence and hesitancy (A/H) in video-based behavioral interventions? These states are often subtle, temporally sparse, and difficult to capture using standard global video representations. The authors aim to improve recognition accuracy by focusing on speech and language cues while explicitly modeling the psycholinguistic markers of uncertainty and conflict.

The researchers propose an audio-text architecture that processes videos as a sequence of overlapping 5-second windows. For each window, the model extracts three types of data: prosodic audio descriptors (e.g., pitch, jitter, shimmer), emotion-oriented text embeddings from RoBERTa, and a 74-dimensional vector of handcrafted psycholinguistic features. These features capture specific indicators of A/H, such as response latency, epistemic hedges, and ambivalence indices based on social psychology frameworks. The model uses temporal cross-attention to fuse audio and text representations and employs gated multiple-instance learning (MIL) to dynamically weight the importance of different windows, allowing the system to focus on the most informative segments of a video.

The integration of psycholinguistic support features provided the most significant performance boost. Compared to a baseline using only audio and text with MIL pooling, adding these support features increased the average precision (AP) from 0.835 to 0.875 and improved the macro-F1 score to 0.722. The authors demonstrate that injecting these features before the MIL pooling stage is more effective than concatenating them later, as it allows the support features to modulate the attention weights assigned to each window.

This study highlights the importance of domain-specific inductive biases in affective computing. By incorporating established psychological theories of uncertainty and ambivalence into the model architecture, the authors achieve better performance than generic deep learning approaches that rely solely on high-dimensional embeddings. This suggests that for tasks involving complex human states, interpretable, handcrafted features can serve as a powerful complement to black-box neural representations.