Varsha Suresh, Mohammad Mahdi Abootorabi, Mohamed Salman, M. Hamza Mughal, Christian Theobalt, Ashwin Ram, Jürgen Steimle, Vera Demberg
4 min
Abstract
Learning a shared representation between spoken text and gesture is central to co-speech gesture retrieval, synthesis, and understanding, but remains challenging for semantically meaningful gestures whose communicative intent is not captured by motion alone. Direct contrastive alignment between transcripts and continuous motion embeddings often overemphasizes low-level kinematics and misses the symbolic content of semantic gestures. We propose semantic motion anchors, natural-language abstractions of gesture motion capturing physical form and communicative intent. Our method discretizes 3D gestures into body-hand motion primitives, verbalizes them into structured descriptions, and grounds them in the transcript to provide auxiliary contrastive supervision. On BEAT2, our method improves text-to-gesture R@1 by 8.2% over a direct text-motion baseline and outperforms prior retrieval approaches on text to gesture and gesture to text retrieval directions. Beyond aggregate retrieval metrics, semantic motion anchor supervision helps retrieve gestures that are semantically meaningful for the spoken query, rather than defaulting to generic motion patterns. A downstream retrieval-augmented gesture generation study showed that users significantly preferred gestures retrieved by our approach over a retrieval-augmented generation baseline, demonstrating that semantically grounded retrieval translates to gestures that better convey communicative intent in downstream generation.
Alex: Here's something I'm wondering though—is there a risk that the AI just *invents* a meaning that isn't really there? Like, it hallucinates an intent?
Sam: That's a fair concern. The researchers addressed it by using what they call an 'LLM-as-judge'—essentially, a separate AI that compares the machine-generated descriptions against ones written by human experts. They found a strong correlation between the two, which suggests the system is genuinely learning the logic of the gesture, not just guessing.
Alex: So the machine's interpretation and the human expert's interpretation largely agree?
Sam: Right. And that matters because it means the descriptions aren't just plausible-sounding—they're actually grounded in what humans recognize as meaningful movement.
Alex: I want to push on one more thing. If you're forcing the model to describe everything in language, doesn't that risk making it too rigid? Like, human gestures are fluid—they're not always neat categories.
Sam: That's a critical point, and the researchers found exactly that tension. If they constrained the model too tightly—giving too much weight to the language descriptions—performance actually dropped. The gestures became stiff and unnatural. They had to find a balance where the language labels *guide* the model without locking it into a fixed template.
Alex: So it's a balancing act between the structure of the language and the flexibility of the motion.
Sam: Precisely. They used something called a 'bridge loss' to maintain that balance. In plain terms, it's a mathematical check that keeps the physical form and the intended meaning from drifting too far apart during training—like a tether that keeps the two sides connected without pulling either one too hard.
Alex: So the key insight isn't just collecting more motion data—it's about *labeling* that data with the right kind of meaning.
Sam: That's the core of it. By forcing the model to describe the form and intent of a movement, you move away from generic hand-waving and toward gestures that actually communicate something. The paper suggests this is a meaningful step toward digital avatars that don't just look human, but gesture like they have something to say.
Alex: It's a useful reminder that for AI to communicate naturally, it needs to understand not just *how* humans move, but *why* they move that way. Thanks for listening to ResearchPod.