Yuxuan Zhang, Haozhong Xiong, Yubo Huang, Jiayi Song, Jinpeng Yu, Haofan Wang, Jiaming Liu, Ruihua Huang, Liwei Wang
5 min
Pose-driven human animation synthesizes a video of a target person from a single reference image and a driving pose stream, with critical applications in live streaming, telepresence, and virtual avatars. However, existing diffusion-based animation systems require minutes to hours of offline computation per clip, completely precluding responsive, interactive communication. Closing this capability gap requires simultaneously solving three major challenges: reducing the inference latency of a billion-scale Diffusion Transformer (DiT), converting a bidirectional video generation model into an open-ended causal generator without quality loss, and preventing identity and appearance drift over arbitrarily long rollouts.
To convert a pretrained bidirectional DiT into a fast causal generator, the authors propose a two-stage training pipeline. In Stage 1, Reference-Anchored Teacher-Forcing Adaptation conditions each generated temporal block on ground-truth clean target history rather than the model's own predictions, while making the reference-image latent globally visible through a permanent Ref Sink. This prevents autoregressive error accumulation during adaptation. In Stage 2, Block-wise Self-Forcing Distillation (BS-DMD) reduces the sampling budget to just three denoising steps. By performing a gradient-free rollout followed by a block-wise replay scheme that optimizes one temporal block at a time, this distillation process exposes every block position to student-induced trajectory distributions without requiring massive memory graphs, making it feasible to train the 14B-parameter model on a single 8x80GB GPU node.
To prevent identity drift during extended streaming without letting memory and computation grow linearly over time, the authors introduce PR-Sink, a bounded KV-cache mechanism. PR-Sink combines a Static Sink that permanently anchors the very first generated block, a three-slot Rolling Window for recent frames, and a Dynamic Sink that retrieves a historical KV block from a compact memory bank based on whole-body pose similarity. When a subject revisits an earlier pose, the dynamic sink restores the relevant appearance and view context from the memory bank. Because the memory bank, sink regions, and rolling window all maintain fixed capacities, memory consumption and per-block latency remain completely constant regardless of how long the stream continues.
Combined with infrastructure optimizations such as Ulysses sequence parallelism and operator fusion to distribute attention computation across GPUs, LiveAnimate achieves 19.63 FPS streaming inference on two NVIDIA H100 GPUs. On a rigorous three-minute long-form evaluation benchmark, the system preserves perceptual quality and identity remarkably well from the first thirty seconds to the final minute. Prior diffusion methods either degrade substantially in visual quality or demand hours of offline processing for the same rollout duration, establishing a new operating point for interactive full-body character animation.
Pose-driven human animation synthesizes a video of a target person from a single reference image and a driving pose stream. Real-time generation is essential for interactive applications such as live streaming, telepresence, and virtual avatars, yet diffusion-based systems require minutes to hours per clip, precluding responsive interaction. We present LiveAnimate, to our knowledge the first animation system to combine real-time streaming with stable long-form generation at billion scale, built on a 14B-parameter video Diffusion Transformer (DiT). A two-stage training pipeline first adapts a pretrained bidirectional DiT into a block-causal autoregressive generator through Reference-Anchored Teacher-Forcing Adaptation, and then reduces the sampling budget to three steps through Block-wise Self-Forcing Distillation. To preserve appearance over extended streams, we introduce Pose-Retrieval Sink Attention (PR-Sink), a bounded KV-cache mechanism combining a Static Sink that permanently anchors the first generated block, a Dynamic Sink that holds a pose-retrieved historical block, and a three-slot Rolling Window. When a pose recurs, PR-Sink restores the relevant appearance context without retaining the entire sequence, so memory and per-block latency remain constant regardless of stream duration. Together with Ulysses sequence parallelism and operator fusion, these designs enable 19.63\,FPS streaming inference on two NVIDIA H100 GPUs. On a three-minute benchmark, LiveAnimate maintains nearly constant perceptual quality and identity from the first 30 seconds to the final minute, while prior systems degrade substantially or require hours of offline computation for the same rollout. These results establish a new operating point in quality, latency, and duration for interactive full-body animation.
Sam: Oh — so if the avatar raises its right arm, the system doesn't just guess what that should look like. It actually retrieves a memory from an earlier moment in the stream when the arm was in a similar position, and uses that as a guide.
Alex: Exactly. The pose acts as a search query. The model finds the closest historical match, retrieves the appearance information from that moment, and uses it to stay consistent. It's a way of having a long memory without actually storing everything.
Sam: That's a genuinely clever workaround. But I want to ask about speed, because this is still a very large model we're talking about. How do they get it running fast enough to feel live?
Alex: They use a two-stage training process. In the first stage, they retrain the model to work within this new block-by-block structure — essentially teaching it the new rules of the road. In the second stage, they apply a technique called distillation. Think of distillation like this: the original model takes many careful steps to produce each frame, the way a painter might layer dozens of brushstrokes. Distillation trains a leaner version of the model to reach a very similar result in far fewer steps — more like a skilled sketch artist who captures the essence quickly.
Sam: So the quality doesn't collapse, it just learns to get there more efficiently.
Alex: That's the intent. The paper reports the system can generate video at around twenty frames per second — which is fast enough to feel continuous and live to a viewer — while keeping the visual quality close to what the original, much slower model would have produced.
Sam: What strikes me about all of this is that none of these solutions are simple shortcuts. The bounded cache, the pose retrieval, the distillation — each one is solving a specific, concrete problem that would otherwise make real-time animation impossible.
Alex: That's a fair summary. What LiveAnimate represents is a careful set of engineering trade-offs: you give up the luxury of seeing the whole video at once, and in return you gain the ability to run indefinitely, in real time, from a single image. Whether that trade-off holds up across a wide range of real-world conditions is something further research will need to examine — but as a proof of concept, the approach is coherent and the reasoning behind each design choice is clear.
Sam: Something to watch as the technology develops.
Alex: Agreed. Thanks for listening to ResearchPod.