We present AnyTalk, a novel method for generating 3D speech animations for arbitrary characters without requiring any animation data. While existing audio-driven 3D speech animation methods rely on character-specific training data or laborious rigging/re-meshing, AnyTalk circumvents these limitations by leveraging recent video diffusion models trained on extensive video datasets. We first adapt a pre-trained video diffusion model to a target character through our Character-specific Fine-tuning (\textit{CsF}) technique. By fine-tuning on rendered images of the 3D character paired with zeroed-out audio embeddings (representing "no motion"), we eliminate the need for animation data while preserving the motion prior of large-scale video diffusion model. We then uplift the resulting talking-head video into a 3D speech animation by estimating blendshape parameters through a proposed optimization process. AnyTalk enables lip-synced animations across diverse face meshes and blendshape configurations, significantly reducing manual effort and data requirements. We further enhance usability by distilling AnyTalk into a streamlined network, $\text{AnyTalk}_{RT}$, thereby enabling real-time performance. By leveraging talking-head video generation, our method broadens access to audio-driven speech animation technology for arbitrary characters. The code is publicly available at https://serin-yoon.github.io/projects/anytalk/.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a method called AnyTalk — a way to animate a 3D character's face to match speech audio, without needing any pre-existing animation data for that character.
Alex: So the core problem is: making a 3D character speak usually requires a huge library of recorded facial movements for that specific character — and most independent creators just don't have that?
Sam: Exactly. Every stylized character has a unique face shape and structure, so standard animation tools trained on one character simply break down when you hand them a different one. AnyTalk sidesteps that by starting from something much simpler — still images.
Alex: How do still images help you animate a face?
Sam: The key insight is that they borrow a model already trained to generate video of people speaking. That model already understands how mouths move during speech. The challenge is teaching it what this particular character looks like — without any video of them talking.
Alex: And how do you teach it that?
Sam: They render a few still images of the 3D character at rest, and pair those images with silent audio — essentially a flat line with no sound. By showing the model "this is what the character looks like when nothing is happening," it learns the character's visual appearance without losing its built-in knowledge of how speech works.
Alex: It's like showing an artist a series of photographs of a puppet and saying "this is what it looks like" — so they can recognize it later, even though it's not moving yet.
Sam: That's a fair way to put it. Once the model has learned the character's appearance, you feed in real speech audio, and it generates a short video of that character's face moving in sync with the words.
Alex: But that video is flat — it's just pixels. How do you get actual 3D movement out of it?
Sam: This is where the geometry step comes in. Think of a 3D character's face as being controlled by a set of invisible sliders — digital artists call these blendshapes. One slider pulls the corners of the mouth into a smile, another drops the jaw, and so on. The system's job is to figure out which combination of slider positions would make the 3D face match each frame of that 2D video.
Alex: So it's working backwards — looking at the video and asking "what slider settings would produce this expression?"
Sam: Precisely. It does that by projecting reference points from the 2D video onto the 3D mesh — think of it like casting a shadow from the video onto the character's face and seeing where the edges land. Then it adjusts those sliders, frame by frame, until the 3D face lines up with what the video shows.
Alex: Does that process happen quickly?
Sam: Not instantly. The computer is solving a careful geometry puzzle for each frame, which takes a few seconds per frame. That's why the authors also built a faster version called AnyTalk RT.
Alex: How does the faster version work?
Sam: They use a technique called model distillation. Imagine a slow but thorough expert solving every puzzle by hand, and then training a faster apprentice to recognize the patterns and skip straight to the answer. The apprentice isn't doing the full calculation — it's learned to predict the right slider values directly from the audio. That brings the processing time down to around nine milliseconds per frame, which is fast enough for live applications like video games or streaming.
Alex: Does the quality take a hit?
Sam: There's a measurable trade-off — the lip-sync accuracy drops slightly compared to the slower method. But for real-time use cases, the speed gain is worth it.
Alex: And they tested this across genuinely different character types — not just variations on a human face?
Sam: Yes, across five distinct avatars with very different structures, including heavily stylized fantasy characters that have no pre-existing animation data at all. The method held up across all of them, because it never assumed a particular face shape to begin with.
Alex: So what are the real-world limits? Where does this approach still fall short?
Sam: The main constraint is that it still depends on the character being properly rigged — meaning someone has already set up those blendshape sliders before the system can use them. And because the video it generates is a single flat view, it can sometimes miss subtle movements that only show up from a different angle. Quick, fleeting expressions can also get smoothed over.
Alex: So the underlying mesh still needs careful preparation. This isn't quite a push-button solution yet.
Sam: Not entirely, no. The authors point to multi-view approaches and more tightly integrated pipelines as the logical next steps — systems that could handle the geometry and the visual synthesis together, rather than as separate stages.
Alex: Still, for independent creators who can't afford motion-capture studios, being able to animate a stylized character from a single image and an audio track is a meaningful shift in what's practically possible.
Sam: That's the central contribution. It moves the barrier from "you need data for every character" to "you need a rigged model and an audio file." That's a significant reduction in what's required to get started.
Alex: Thanks for walking us through it, and thanks to our listeners for joining us on ResearchPod.