AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Model | Kwan Yun et al. | ResearchPod

Transcript: AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Model

Alex: Welcome to another episode of ResearchPod.

Sam: Today we're looking at a method called AnyTalk — a way to animate a 3D character's face to match speech audio, without needing any pre-existing animation data for that character.

Alex: So the core problem is: making a 3D character speak usually requires a huge library of recorded facial movements for that specific character — and most independent creators just don't have that?

Sam: Exactly. Every stylized character has a unique face shape and structure, so standard animation tools trained on one character simply break down when you hand them a different one. AnyTalk sidesteps that by starting from something much simpler — still images.

Alex: How do still images help you animate a face?

Sam: The key insight is that they borrow a model already trained to generate video of people speaking. That model already understands how mouths move during speech. The challenge is teaching it what this particular character looks like — without any video of them talking.

Alex: And how do you teach it that?

Sam: They render a few still images of the 3D character at rest, and pair those images with silent audio — essentially a flat line with no sound. By showing the model "this is what the character looks like when nothing is happening," it learns the character's visual appearance without losing its built-in knowledge of how speech works.

Alex: It's like showing an artist a series of photographs of a puppet and saying "this is what it looks like" — so they can recognize it later, even though it's not moving yet.

Sam: That's a fair way to put it. Once the model has learned the character's appearance, you feed in real speech audio, and it generates a short video of that character's face moving in sync with the words.

Alex: But that video is flat — it's just pixels. How do you get actual 3D movement out of it?

Sam: This is where the geometry step comes in. Think of a 3D character's face as being controlled by a set of invisible sliders — digital artists call these blendshapes. One slider pulls the corners of the mouth into a smile, another drops the jaw, and so on. The system's job is to figure out which combination of slider positions would make the 3D face match each frame of that 2D video.

Alex: So it's working backwards — looking at the video and asking "what slider settings would produce this expression?"

Sam: Precisely. It does that by projecting reference points from the 2D video onto the 3D mesh — think of it like casting a shadow from the video onto the character's face and seeing where the edges land. Then it adjusts those sliders, frame by frame, until the 3D face lines up with what the video shows.

AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Model

Kwan Yun, Serin Yoon, Sunjin Jung, Jung Eun Yoo, Inyup Lee, Junyong Noh

Aug 17, 2026·4:57·arxiv·Original Paper

Audio-driven 3D Facial AnimationCharacter-specific Fine-tuning (CsF)Video Diffusion ModelsBlendshape ParametersAnyTalkBlendshape Optimization

Listen

5 min

Briefing

Introduction

Generating realistic 3D speech animation across diverse characters typically requires laborious manual keyframing or extensive character-specific training data. While deep learning methods reduce manual effort, they usually depend on paired audio-3D data for each unique mesh structure or blendshape configuration. This requirement poses a significant barrier for independent developers and small studios. Meanwhile, recent video diffusion models trained on massive video datasets can generate highly realistic talking-head videos from unseen characters, but directly applying them to 3D avatars introduces severe domain gaps, such as visual inconsistencies and unnatural head motion.

To bridge this gap, the authors present AnyTalk, a novel framework that leverages a 2D video generation model to produce 3D speech animation for arbitrary characters without needing any 3D animation data. The system decouples the process into two main stages: generating a character-aligned talking-head video from input audio, and uplifting that video into a 3D speech animation via blendshape optimization.

Character-Specific Fine-Tuning and Video Generation

Directly using pre-trained video generation models on 3D character renders leads to visual mismatches, unwanted head movements, and artifacts like altered clothing or distorted teeth. To resolve this, AnyTalk introduces Character-specific Fine-tuning (CsF). By rendering frontal images of the 3D character for individual active blendshapes and pairing them with zeroed-out audio embeddings representing no motion, the model learns the spatial appearance of the target character without forgetting how to animate. During training, all attention layers and reference networks are frozen while only the spatial residual network of the denoising UNet is updated.

During inference, the fine-tuned model receives non-linear, non-zero speech audio. By utilizing specialized scaling factors for pose, expression, and lip control weights (such as zeroing out head rotation while amplifying lip motion), the model generates dynamic, lip-synced facial movements while maintaining a completely static head pose. This stable head position significantly simplifies the subsequent 3D optimization phase.

Blendshape Optimization and Real-Time Distillation

Once the 2D talking-head video is generated, AnyTalk estimates the underlying blendshape parameters through an optimization process. First, landmarks are extracted from a rendered neutral image of the character, and rays are cast to identify corresponding mesh vertices. The optimization objective then minimizes a talk landmark loss (enhanced with an image homography using talk-invariant landmarks to handle subtle head shifts), an asymmetric mouth opening loss to prevent under-articulation, and a regularization loss to prevent changes to unrelated facial features like eyebrows.

To improve usability for real-time applications, the authors distill AnyTalk into a streamlined network called AnyTalk RT. Using feature matching and reconstruction losses, this distilled variant achieves an inference speed of 110 frames per second, making it highly practical for interactive games and virtual reality platforms.

Key Terms and Definitions

Blendshape Parameters — Numerical coefficients that control the deformation of a 3D character mesh to produce specific facial expressions and phoneme configurations.
Character-specific Fine-tuning (CsF) — A personalization technique that adapts a general video diffusion model to a target 3D character using still renders paired with zeroed-out audio embeddings, avoiding the need for training animation data.
Talk-Invariant Landmarks — Facial landmarks that do not shift during speech articulation, used to compute robust image homography and alignment between the rendered character and the generated video.
Zeroed-Out Audio Embedding — A modified audio input vector representing no motion, used during fine-tuning to disentangle static appearance reproduction from speech-driven motion signals.
Ray-Casting Vertex Selection — A process of casting rays from 2D facial landmarks onto a rendered neutral 3D character image to automatically identify and map corresponding 3D mesh vertices.

Abstract

We present AnyTalk, a novel method for generating 3D speech animations for arbitrary characters without requiring any animation data. While existing audio-driven 3D speech animation methods rely on character-specific training data or laborious rigging/re-meshing, AnyTalk circumvents these limitations by leveraging recent video diffusion models trained on extensive video datasets. We first adapt a pre-trained video diffusion model to a target character through our Character-specific Fine-tuning (\textit{CsF}) technique. By fine-tuning on rendered images of the 3D character paired with zeroed-out audio embeddings (representing "no motion"), we eliminate the need for animation data while preserving the motion prior of large-scale video diffusion model. We then uplift the resulting talking-head video into a 3D speech animation by estimating blendshape parameters through a proposed optimization process. AnyTalk enables lip-synced animations across diverse face meshes and blendshape configurations, significantly reducing manual effort and data requirements. We further enhance usability by distilling AnyTalk into a streamlined network, $\text{AnyTalk}_{RT}$, thereby enabling real-time performance. By leveraging talking-head video generation, our method broadens access to audio-driven speech animation technology for arbitrary characters. The code is publicly available at https://serin-yoon.github.io/projects/anytalk/.

AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Model: Research Paper Podcast