TRACE: Temporal Relationship-Aware Conversational Entrainment Detection in Dyadic Speech

Understanding how speakers align their emotional states—a process known as emotional entrainment—is critical for developing responsive speech AI agents. However, measuring this alignment is difficult because it depends heavily on the social relationship between speakers and the specific conversational context. The authors introduce the Dyadic Emotional Entrainment Dataset (DyadEE) and the Temporal Relationship-Aware Conversational Entrainment Detection (TRACE) framework to address this challenge.

To overcome the lack of large-scale annotated data, the authors construct DyadEE by taking natural, entrained conversations and creating synthetic 'non-entrained' counterparts. They use emotion-contradicting resynthesis to disrupt affective coordination while maintaining acoustic realism. To prevent models from simply learning to detect synthetic artifacts, they also include voice-converted entrained dyads, forcing the model to focus on emotional dynamics rather than audio quality.

TRACE models dyadic interaction as a temporal sequence of alternating speaker windows. By using an emotion-fine-tuned Whisper encoder, the model extracts acoustic embeddings that capture prosodic and affective cues. These are processed through a stack of LLaMA-style blocks using bidirectional self-attention, allowing the model to analyze how emotional states evolve across the entire conversation. The framework incorporates context and relationship information via two distinct architectures: Late Fusion (TRACE-LF) and Cross-Encoder (TRACE-CE).

Experimental results demonstrate that incorporating relational and contextual signals significantly boosts detection performance. TRACE-CE achieved the highest accuracy (93.47%) when conditioned on relationship information. The authors found that the two conditioning signals are architecture-dependent: TRACE-LF benefits most from context, while TRACE-CE excels when provided with relationship data. Human evaluations confirmed that the dataset's binary labels correspond to perceptible differences in emotional coordination, validating the use of controlled disruption as a proxy for studying real-world entrainment.