Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, yet their responses remain visually disembodied. We introduce \textbf{Ex-Omni-2D}, an omni-modal dialogue framework that generates a coordinated response comprising text, personalized speech, and reference-conditioned video. Given a multimodal query, reference image, and reference audio, the model predicts a structured \textit{Visual Thought Plan} (VTP) describing scene, emotion, and motion, followed by response text and native multi-codebook speech units. These units form a shared acoustic-temporal interface: they are decoded into speech and aligned online with video frames. This interface enables the response and avatar pathways to be learned from heterogeneous speech, dialogue, and avatar-video data, avoiding the need for large-scale query--text--speech--video supervision. A full-sequence Video Generator serves as the primary Teacher. For efficient incremental generation, we further distill it into a few-step block-causal \emph{Streaming Student} whose Prefix Streaming mechanism carries a clean latent across consecutive chunks to reduce cumulative late-chunk degradation. With four-step inference, the complete four-GPU pipeline achieves an end-to-end RTF of 1.293 at $400\times720$/$720\times400$, providing a practical quality--efficiency operating point.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a new way to give artificial intelligence a face. Sam, what's the core puzzle this paper is trying to solve?
Sam: We're discussing a framework called Ex-Omni-2D. The challenge is that while modern AI can hold a conversation, it usually lacks any physical presence. This research creates a digital agent that doesn't just speak — it generates a video response where the avatar's facial expressions and movements are perfectly synchronized with what it's saying.
Alex: So, this is about moving beyond a disembodied voice to a truly expressive, visual agent?
Sam: Exactly. Current systems struggle to link what they say with how they look. If you ask a virtual assistant a complex question, you want it to look thoughtful, or smile, in a way that matches the tone of the answer. This paper proposes a way to plan that visual behavior directly from the conversation itself.
Alex: That sounds like a lot of moving parts. How do they coordinate speech with video without it looking like a disconnected animation?
Sam: They use what they call a "Visual Thought Plan." Think of it like a film director writing a script that includes both the dialogue and the stage directions — like "look concerned" or "smile warmly" — before the actors even step on set. By planning those visual details as a structured sequence first, the model ensures the avatar's behavior actually matches what's being said.
Alex: So the model "thinks" about the visual cues before it starts generating the video. But how does it keep the lip movements and the speech perfectly in sync?
Sam: That's where something called multi-codebook speech units come in. Imagine the voice is broken down into tiny, discrete digital building blocks — almost like individual Lego pieces that each represent a specific sound. Because the same set of building blocks is used both to generate the audio and to tell the video generator how to move the mouth, the two stay perfectly aligned. There's no guesswork involved.
Alex: Like using the same sheet music for both the singer and the conductor. But what happens if the conversation runs long? Doesn't the system lose track of the character?
Sam: That's a common hurdle in real-time AI. To solve it, they use something called Prefix Streaming. Instead of generating each short chunk of video — say, three seconds — completely from scratch, the system carries over a small, clean piece of information from the previous chunk. Think of it like a relay race where the baton passes a little context forward. That handoff helps the model stay consistent, so the character doesn't suddenly look different or glitch out mid-response.
Alex: So it's about efficiency and continuity working together. It sounds like they've found a practical balance between high-quality video and the speed a real-time conversation actually demands.
Sam: That is the primary finding. The system works by distilling a large, slow model — the "teacher" — into a faster, leaner one called the "student." The student learns to mimic the teacher's quality while using the streaming approach to stay fast enough for live interaction. It's worth noting that the model's pure reasoning scores do drop slightly in the process, since it's now dividing its attention between thinking and planning visual behavior simultaneously.
Alex: So there's a real trade-off there — you gain expressiveness, but you give up a small amount of raw reasoning ability.
Sam: Precisely. And that's an honest tension the paper acknowledges. The goal isn't to replace a powerful reasoning model — it's to create something that feels genuinely present in a conversation. Whether that trade-off is worth it depends entirely on what you're building the agent for.
Alex: That's a useful distinction to hold onto. If you're building a customer service agent or a digital companion, the expressiveness probably matters more than raw problem-solving. If you're building a research assistant, maybe not.
Sam: Exactly. And that's what makes this work interesting rather than universal — it's a clear step forward for a specific class of applications where presence and naturalness matter. The framework is modular enough that future work could potentially push both sides of that trade-off further.
Alex: Thanks for walking us through it, Sam. And thanks to everyone listening to ResearchPod.