Real-time Generation of Listener Nodding via Prediction of Kinematic Parameters for Avatar Dialogue Systems

This study addresses the challenge of making conversational avatars more human-like by generating context-aware nonverbal feedback. While many existing systems can predict when a listener should nod, they often rely on fixed, repetitive motion patterns. The researchers developed a two-module architecture: a timing prediction module and a kinematic parameter prediction module. Both modules use a dyadic attention network based on Voice Activity Projection (VAP) to process the speech signals of both the speaker and the listener simultaneously. By using continuous features from a neural audio encoder (Mimi) rather than discrete tokens, the model maintains causality and captures nuanced acoustic and semantic information to inform its predictions.

The proposed model significantly outperforms baseline methods—including those using stochastic timing or fixed-motion nodding—in both timing accuracy and the naturalness of the generated motion. A key innovation is the use of multi-task learning: the kinematic parameter module is initialized from the timing-trained module and then fine-tuned. This approach allows the system to predict specific motion features like the number of nodding cycles, the magnitude of the head movement (range), and the velocity (speed) based on the ongoing dialogue context. Subjective evaluations confirm that this method produces more human-like and contextually appropriate listener responses.

Effective nonverbal communication is essential for smooth human-computer interaction. By moving beyond simple binary "nod/no-nod" predictions to generating diverse, context-sensitive head movements, this system enables avatars to better demonstrate active listening. The model is lightweight and capable of real-time operation, making it a practical tool for integration into interactive spoken dialogue systems and virtual agents.