Jintao Zhang, Kai Jiang, Jintao Chen, Xu Wang, Yang Luo, Yuji Wang, Dechuang Chen, Jungang Li, Chengyang Ye, Marco Chen, Hongzhou Zhu, Min Zhao, Yuxuan Jiang, Zhengkun Huang, Chendong Xiang, Kaiwen Zheng, Haoxu Wang, Xiaohang Wang, Qi Jia, Xin Chen, Yimin Chen, Youhe Jiang, Fangcheng Fu, Zhijie Deng, Fan Bao, Jianfei Chen, Jun Zhu
6 min
Abstract
We introduce Vidu S1, a real-time interactive video generation model supporting voice control of digital characters. Users can control video generation content at any moment through voice instructions. Vidu S1 supports infinite-length real-time video generation without blurring, drift, or visual distortion. Built with TurboDiffusion and TurboServe, Vidu S1 outputs 540p real-time videos at up to 42 FPS on regular consumer GPUs. Users can upload custom images of real people, anime, and pets, and choose different voice tones for personalized experiences. Experiments show that Vidu S1 achieves the best performance across all test metrics while fully meeting real-time inference requirements. A playable online demo is available at https://vidu.com/vidu-stream.
Sam: To make all of this work, the model must need very high-quality footage to learn from. How do they make sure the training data isn't messy?
Alex: They built a large filtering pipeline. Raw video is run through multiple stages to remove watermarks, blurry footage, or clips where the audio and video don't match up. They use a mix of specialised models for technical checks, and a broader model to evaluate overall quality and tone.
Sam: So it's like having two reviewers—one checking for technical glitches, and another stepping back to ask whether the clip is actually good.
Alex: Exactly. They also filter out clips with multiple speakers, because that makes it much harder for the model to learn how to sync lip movements with a single voice. And they add captions at two levels—one describing the whole clip, and another describing specific moments—so the model learns exactly what action happens at what time.
Sam: So the data pipeline is doing a lot of heavy lifting before training even begins. How does the training itself work?
Alex: It happens in three stages. In the first stage, they train the model to see an entire video clip at once—beginning, middle, and end simultaneously. The goal is to learn the visual patterns of a complete, high-quality sequence. Think of it as studying the full script of a film before you try to act in it.
Sam: But in a live stream, you don't have the "end" yet. How do they make the jump from that full-picture learning to working in real time?
Alex: That's the second stage. They switch the model to a different mode where it can only use what has already happened to predict the next frame—no peeking at the future. It has to work the way a live stream actually works.
Sam: And that's where "Diffusion Forcing" comes in?
Alex: Right. In standard training, the model always sees perfect, clean images from the past. But in a real stream, it will inevitably make small mistakes. If it has only ever practised with perfect data, it won't know how to recover when things go slightly wrong.
Sam: So it's like a footballer who only ever practises on a perfect pitch—the first time the ground is muddy, they're lost?
Alex: That's a good way to put it. Diffusion Forcing intentionally feeds the model slightly noisy, imperfect versions of past frames during training. It learns to correct its own errors on the fly, so small mistakes don't spiral into something unrecoverable.
Sam: And the third stage is about speed?
Alex: Yes. Even with a well-trained model, generating video in real time requires the underlying calculations to be very fast. The third stage compresses the complex, multi-step process the model uses to build each image into a much smaller number of efficient steps, without losing the quality of the output.
Sam: So the three stages are: learn what good video looks like, learn to work with only the past, and then streamline the maths so it runs fast enough for a live user.
Alex: That's it. Each stage addresses a different problem. Accuracy, resilience to error, and speed. Without all three, you'd have a model that either drifts, freezes up, or can't keep pace with a real conversation.
Sam: Is keeping video from drifting really the central technical problem in AI video right now?
Alex: It is one of the most significant ones. When you generate frame by frame, every tiny error compounds. Without those specific training stages, the system would lose its visual identity within seconds of a long generation.
Sam: So it's not just about producing a good-looking image. It's about keeping that image coherent while it's moving, changing, and responding to someone in real time.
Alex: Precisely. The goal is for the AI to feel like a persistent, reliable presence in a conversation—not a flickering, unpredictable image that slowly stops resembling itself. And the combination of the TwinCache memory system, the three-stage training process, and the optimised serving infrastructure is what the paper argues makes that possible.
Sam: That's a genuinely interesting set of engineering choices. Each one solving a specific failure mode rather than just throwing more computing power at the problem.
Alex: Which is often where the most durable progress comes from. Thanks for listening to ResearchPod.