We introduce Vidu S1, a real-time interactive video generation model supporting voice control of digital characters. Users can control video generation content at any moment through voice instructions. Vidu S1 supports infinite-length real-time video generation without blurring, drift, or visual distortion. Built with TurboDiffusion and TurboServe, Vidu S1 outputs 540p real-time videos at up to 42 FPS on regular consumer GPUs. Users can upload custom images of real people, anime, and pets, and choose different voice tones for personalized experiences. Experiments show that Vidu S1 achieves the best performance across all test metrics while fully meeting real-time inference requirements. A playable online demo is available at https://vidu.com/vidu-stream.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a model called Vidu S1, which is designed to turn AI video generation into a live, interactive experience.
Sam: So this paper is asking how we move past the current "type and wait" approach—where you describe what you want, and the computer goes off and builds the whole video before you see anything?
Alex: Exactly. Most current models work like an offline render. You submit a request, the computer spends minutes building the entire video, and you can't change your mind or give new instructions while it's working.
Sam: And that "one-shot" approach breaks down completely for anything live—streaming, real-time conversation, that kind of thing?
Alex: That's the central challenge. The team behind Vidu S1 wanted to shift from that passive, offline style to an active system where you can use your voice to guide a character as it moves, in real time.
Sam: That sounds like a meaningful step. But how do they handle "drift"? I've seen AI videos where a character starts to look strange after just a few seconds of motion.
Alex: That happens because of how these models predict the next frame. They build a sequence step by step, and if there's a tiny error in one frame, the next frame uses that error as its starting point. The mistake grows with every step.
Sam: Like a game of telephone, where the message gets more garbled the longer it goes?
Alex: Exactly. To fix this, they use what they call a "TwinCache" strategy. Imagine a sketch artist who keeps two versions of their work side by side: a rough, quick gesture sketch that captures motion, and a clean, detailed ink outline that preserves the character's identity.
Sam: So one cache handles the movement, and the other keeps the character looking like themselves?
Alex: Right. By splitting those two jobs, the system can stay fluid without the character's face or clothing slowly drifting into something unrecognisable over time.
Sam: And they claim this works indefinitely? How do they stop the computer from slowing down as the video gets longer and longer?
Alex: They use a "sliding window" approach. Instead of trying to remember every single frame from the very start, the model only keeps a fixed, recent window of time in active memory—plus a persistent reference that reminds it what the character is supposed to look like.
Sam: So it's like only keeping the last few seconds of memory active, while holding onto a "master copy" of the character as a constant anchor?
Alex: Precisely. That's what allows the system to generate video without processing an ever-growing pile of past data.
Sam: To make all of this work, the model must need very high-quality footage to learn from. How do they make sure the training data isn't messy?
Alex: They built a large filtering pipeline. Raw video is run through multiple stages to remove watermarks, blurry footage, or clips where the audio and video don't match up. They use a mix of specialised models for technical checks, and a broader model to evaluate overall quality and tone.
Sam: So it's like having two reviewers—one checking for technical glitches, and another stepping back to ask whether the clip is actually good.
Alex: Exactly. They also filter out clips with multiple speakers, because that makes it much harder for the model to learn how to sync lip movements with a single voice. And they add captions at two levels—one describing the whole clip, and another describing specific moments—so the model learns exactly what action happens at what time.
Sam: So the data pipeline is doing a lot of heavy lifting before training even begins. How does the training itself work?
Alex: It happens in three stages. In the first stage, they train the model to see an entire video clip at once—beginning, middle, and end simultaneously. The goal is to learn the visual patterns of a complete, high-quality sequence. Think of it as studying the full script of a film before you try to act in it.
Sam: But in a live stream, you don't have the "end" yet. How do they make the jump from that full-picture learning to working in real time?
Alex: That's the second stage. They switch the model to a different mode where it can only use what has already happened to predict the next frame—no peeking at the future. It has to work the way a live stream actually works.
Sam: And that's where "Diffusion Forcing" comes in?
Alex: Right. In standard training, the model always sees perfect, clean images from the past. But in a real stream, it will inevitably make small mistakes. If it has only ever practised with perfect data, it won't know how to recover when things go slightly wrong.
Sam: So it's like a footballer who only ever practises on a perfect pitch—the first time the ground is muddy, they're lost?
Alex: That's a good way to put it. Diffusion Forcing intentionally feeds the model slightly noisy, imperfect versions of past frames during training. It learns to correct its own errors on the fly, so small mistakes don't spiral into something unrecoverable.
Sam: And the third stage is about speed?
Alex: Yes. Even with a well-trained model, generating video in real time requires the underlying calculations to be very fast. The third stage compresses the complex, multi-step process the model uses to build each image into a much smaller number of efficient steps, without losing the quality of the output.
Sam: So the three stages are: learn what good video looks like, learn to work with only the past, and then streamline the maths so it runs fast enough for a live user.
Alex: That's it. Each stage addresses a different problem. Accuracy, resilience to error, and speed. Without all three, you'd have a model that either drifts, freezes up, or can't keep pace with a real conversation.
Sam: Is keeping video from drifting really the central technical problem in AI video right now?
Alex: It is one of the most significant ones. When you generate frame by frame, every tiny error compounds. Without those specific training stages, the system would lose its visual identity within seconds of a long generation.
Sam: So it's not just about producing a good-looking image. It's about keeping that image coherent while it's moving, changing, and responding to someone in real time.
Alex: Precisely. The goal is for the AI to feel like a persistent, reliable presence in a conversation—not a flickering, unpredictable image that slowly stops resembling itself. And the combination of the TwinCache memory system, the three-stage training process, and the optimised serving infrastructure is what the paper argues makes that possible.
Sam: That's a genuinely interesting set of engineering choices. Each one solving a specific failure mode rather than just throwing more computing power at the problem.
Alex: Which is often where the most durable progress comes from. Thanks for listening to ResearchPod.