Kwan Yun, Serin Yoon, Sunjin Jung, Jung Eun Yoo, Inyup Lee, Junyong Noh
5 min
Abstract
We present AnyTalk, a novel method for generating 3D speech animations for arbitrary characters without requiring any animation data. While existing audio-driven 3D speech animation methods rely on character-specific training data or laborious rigging/re-meshing, AnyTalk circumvents these limitations by leveraging recent video diffusion models trained on extensive video datasets. We first adapt a pre-trained video diffusion model to a target character through our Character-specific Fine-tuning (\textit{CsF}) technique. By fine-tuning on rendered images of the 3D character paired with zeroed-out audio embeddings (representing "no motion"), we eliminate the need for animation data while preserving the motion prior of large-scale video diffusion model. We then uplift the resulting talking-head video into a 3D speech animation by estimating blendshape parameters through a proposed optimization process. AnyTalk enables lip-synced animations across diverse face meshes and blendshape configurations, significantly reducing manual effort and data requirements. We further enhance usability by distilling AnyTalk into a streamlined network, $\text{AnyTalk}_{RT}$, thereby enabling real-time performance. By leveraging talking-head video generation, our method broadens access to audio-driven speech animation technology for arbitrary characters. The code is publicly available at https://serin-yoon.github.io/projects/anytalk/.
Alex: Does that process happen quickly?
Sam: Not instantly. The computer is solving a careful geometry puzzle for each frame, which takes a few seconds per frame. That's why the authors also built a faster version called AnyTalk RT.
Alex: How does the faster version work?
Sam: They use a technique called model distillation. Imagine a slow but thorough expert solving every puzzle by hand, and then training a faster apprentice to recognize the patterns and skip straight to the answer. The apprentice isn't doing the full calculation — it's learned to predict the right slider values directly from the audio. That brings the processing time down to around nine milliseconds per frame, which is fast enough for live applications like video games or streaming.
Alex: Does the quality take a hit?
Sam: There's a measurable trade-off — the lip-sync accuracy drops slightly compared to the slower method. But for real-time use cases, the speed gain is worth it.
Alex: And they tested this across genuinely different character types — not just variations on a human face?
Sam: Yes, across five distinct avatars with very different structures, including heavily stylized fantasy characters that have no pre-existing animation data at all. The method held up across all of them, because it never assumed a particular face shape to begin with.
Alex: So what are the real-world limits? Where does this approach still fall short?
Sam: The main constraint is that it still depends on the character being properly rigged — meaning someone has already set up those blendshape sliders before the system can use them. And because the video it generates is a single flat view, it can sometimes miss subtle movements that only show up from a different angle. Quick, fleeting expressions can also get smoothed over.
Alex: So the underlying mesh still needs careful preparation. This isn't quite a push-button solution yet.
Sam: Not entirely, no. The authors point to multi-view approaches and more tightly integrated pipelines as the logical next steps — systems that could handle the geometry and the visual synthesis together, rather than as separate stages.
Alex: Still, for independent creators who can't afford motion-capture studios, being able to animate a stylized character from a single image and an audio track is a meaningful shift in what's practically possible.
Sam: That's the central contribution. It moves the barrier from "you need data for every character" to "you need a rigged model and an audio file." That's a significant reduction in what's required to get started.
Alex: Thanks for walking us through it, and thanks to our listeners for joining us on ResearchPod.