Wan Team
10 min
Wan-Streamer is an end-to-end foundation model designed for real-time, full-duplex interaction. Unlike traditional systems that chain together separate modules for voice activity detection (VAD), speech recognition (ASR), text-to-speech (TTS), and video rendering, Wan-Streamer integrates perception, reasoning, and generation into a single Transformer architecture. By treating language, audio, and video as interleaved tokens in a unified causal sequence, the model maintains a persistent interaction state that allows for natural, low-latency responses, including synchronized speech and visual motion.
The model is built on a strictly causal design to ensure streamability. It uses causal encoders and decoders alongside block-causal attention to process inputs incrementally. During inference, the team employs a "thinker-performer" pipeline to maximize hardware efficiency. The thinker handles perception, state updates, and output decoding, while the performer focuses on the computationally intensive task of latent generation via flow matching. This separation allows the system to overlap perception, generation, and communication across adjacent streaming units, achieving a model-side response latency of approximately 200 ms.
Most current interactive systems suffer from "cascaded latency," where waiting for one module to finish before the next begins leads to sluggish, unnatural interactions. By unifying these processes, Wan-Streamer avoids the error accumulation and synchronization delays inherent in modular pipelines. This approach enables sub-second full-duplex communication, which is critical for applications like embodied AI assistants, live digital humans, and interactive entertainment where responsiveness and long-term dialogue consistency are essential.
We present Wan-Streamer, a native-streaming, end-to-end interactive foundation model designed from the ground up for real-time, low-latency, full-duplex audio-visual interaction. Wan-Streamer seamlessly models language, audio, and video as both input and output within a single Transformer, where the sequence is represented as interleaved visual, audio, and text input tokens together with visual, audio, and text output tokens, coordinated by block-causal attention for incremental streaming. Unlike cascaded interactive systems that rely on separate VAD, ASR, language, TTS, audio-driven animation, or video-generation modules, Wan-Streamer does not rely on external language, speech, avatar, or video-generation modules: perception, reasoning, generation, response timing, turn management, and cross-modal synchronization are learned jointly within one unified model, reducing pipeline latency and error accumulation. To support natural audio-visual responsiveness, we redesign the entire stack around streamability, including causal encoders, causal decoders, block-causal attention, and low-latency multimodal token scheduling, enabling streaming units as short as 160 ms at 25 fps. Wan-Streamer achieves approximately 200 ms model-side response latency and approximately 550 ms total interaction latency when combined with 350 ms bidirectional network latency, supporting sub-second duplex audio-visual communication. These results position Wan-Streamer as a unified, end-to-end, multimodal interactive foundation model for low-latency streaming interaction.
Alex: That's fast. It sounds like the key insight is that it never stops paying attention to the user, even while it's busy generating its own response.
Sam: Precisely. Because the system treats the entire interaction as one continuous, shared history, it can adjust its own behavior mid-sentence. If you start talking, your voice and face appear in that same shared stream, and the model adapts immediately. It's a meaningful shift from systems that are essentially deaf while they're producing output.
Alex: So the bottleneck wasn't just the speed of the hardware—it was the way the software was structured in the first place.
Sam: That is the main takeaway. By designing the system to only ever move forward in time and never rely on future data, they've built something that feels much more like a natural, real-time conversation. It's a notable step toward making digital agents feel genuinely responsive rather than just reactive.
Alex: I want to dig into that Thinker-Performer split a bit more. If the model is one big continuous stream, how does it physically divide the work without losing the thread of the conversation?
Sam: Think of it like a relay team where the runners never stop moving. The Thinker is the strategist—it keeps the long-term memory of what's been said and updates the current state of the conversation. The Performer is the worker—it handles the complex calculations needed to turn those abstract ideas into actual audio and video. Crucially, they share a kind of digital notebook that stores all the important context from the conversation so far. Instead of re-reading the entire history every time it needs to generate a new frame, the system just passes this notebook between the two roles.
Alex: But if they're running at the same time, how do they stay in sync?
Sam: That shared notebook is the key. While the Performer is busy creating the next few milliseconds of video, the Thinker is already looking at what you just said and updating the notebook for the step after that. It's a bit like a chef who chops vegetables for the next dish while the current one is still simmering on the stove. Neither task has to wait for the other to fully finish before it can make progress.
Alex: And how does it turn those internal representations into actual sound and movement?
Sam: They use a technique for generating output that works by starting with something messy—like a blurry photo or garbled audio—and mathematically guiding it toward a clear, clean result. Imagine having a map that tells you exactly how to move from a random blur into a sharp image, step by step. By applying this to compressed mathematical representations of audio and video, the model can generate smooth, lifelike responses in real time. It's a meaningful improvement over older methods that often produced jerky or repetitive movements.
Alex: It sounds like the whole design is focused on removing any reason for the AI to pause and collect itself.
Sam: That is the goal. By integrating everything into one pipeline and overlapping the work, the researchers achieved a response time of around two hundred milliseconds on the model's side. Even accounting for network delay, the total interaction is fast enough to feel close to instantaneous. It's a meaningful shift in how we approach building agents that need to keep pace with human conversation.
Alex: You mentioned the training process happens in three distinct stages. How do they actually teach a model to handle audio, video, and text all at once?
Sam: It's a progression. First, they train the model on large datasets of images, audio, and video separately—think of it like teaching a student the basics of math, reading, and art before asking them to tackle a complex real-world project. Crucially, they align these different types of information so the model learns that a specific sound often matches a specific visual action. It's building a foundational understanding of how different senses relate to each other.
Alex: So it's not just learning to transcribe speech or generate images in isolation—it's learning the connections between them?
Sam: Exactly. The second stage introduces what they call interaction training. The model is fed interleaved streams—text, audio, and video all at once—drawn from real conversational data. It learns to update its internal state based on what you're doing right now, while also planning what it should say or show next. It's learning the rhythm of a conversation, not just the content.
Alex: And that covers the active listening part? Does it naturally learn when to pause or adjust?
Sam: Yes, because the model sees the user's inputs and its own outputs in the same continuous stream, it learns that if a user starts to speak, it should adapt its own output. It's not following a hard-coded rule. It's learned that behavior because it's part of the natural flow of the data it was trained on. The final stage is about speed. They take a large, capable model and compress its knowledge into a smaller, faster version—like distilling a thick reference book into a concise field guide that a practitioner can actually carry around and use quickly.
Alex: So the big model is the expert, and the smaller model learns to approximate that expertise fast enough for a real conversation?
Sam: That's a fair way to put it. They use a technique called rolling distillation to keep the smaller model consistent during long conversations—it practices on its own generated history, which helps it stay on track over time. By the end of training, the model has been shaped to be both capable enough to understand you and fast enough to react without that robotic delay we usually associate with AI assistants.
Alex: It sounds like the training is just as important as the architecture. They're not just building a faster engine—they're teaching it how to behave.
Sam: That's the core of the research. By aligning perception, reasoning, and generation into one sequence, they've created something that doesn't just process data—it participates in a conversation.
Alex: We've talked about how this model functions, but I'm curious about the limitations. If this approach is a clear improvement, why isn't it already powering every digital assistant we use?
Sam: That is a fair question. The main limitation right now is visual resolution. In this initial version, the visual output is limited to a fairly low resolution—think of watching a video on a very early mobile phone screen. It's enough to see expressions and reactions, but it's far from high-definition.
Alex: So it's a proof of concept? They're validating the architecture before worrying about making it look photorealistic?
Sam: Exactly. The researchers focused on proving that the streaming logic works—the ability to listen, think, and speak without lag—rather than on visual quality. Scaling to higher resolution is an engineering challenge, but it requires significantly more computing power. The current model demonstrates that the end-to-end approach is viable for real-time interaction.
Alex: It's like building the engine of a car first to prove it can hit the speed limit, before you worry about the interior.
Sam: That's a good way to put it. By proving that a single unified system can handle the full duplex flow—where the AI and user interact simultaneously—they've cleared the most significant architectural hurdle. You don't need a chain of separate modules to make an agent feel responsive.
Alex: If they scale this up, where does it lead?
Sam: The implications are significant. We could move toward interfaces that feel like a real human presence—telepresence, robotics, or virtual assistants that don't just wait for you to finish a sentence, but react to your tone and expressions in the moment. The research suggests that for AI to feel natural, it must be designed from the ground up as a native full-duplex system, rather than stitching together separate modules after the fact.
Alex: It sounds like the goal is to make the technology disappear, so you're just having a conversation.
Sam: That is the ultimate aim. We're moving away from modular pipelines and toward agents that perceive and express themselves in one unified, human-like rhythm. Whether that vision fully materialises depends on solving the remaining engineering challenges—resolution, computing cost, and data efficiency among them. But this paper makes a credible case that the underlying approach is sound.
Alex: That's a thoughtful place to leave it. Thanks for walking us through it, Sam, and thanks to everyone listening to ResearchPod.