We present MOSS-VL, an open vision-language model family that treats real-time interaction -- perceiving while it speaks -- as a first-class capability. It is co-designed across the stack: the language decoder attends to vision only through gated cross-attention, so the model can naturally see incoming frames while generating; a synthesized interaction corpus supervises when to speak, when to stay silent, and when to revise; and a staged curriculum concentrates all real-time-specific training in one light final stage over a strong offline foundation. Offline, MOSS-VL-Instruct is competitive at comparable scale and leads temporal-reasoning video sets. Across four streaming benchmarks, MOSS-VL-Realtime posts the best average on three (second on the fourth) among open-source streaming models, sweeping the three subsets that squarely test proactive behavior -- 66.0 vs. 37.5 for the best baseline on OmniMMI Proactive Alerting. With 11.3B parameters but visual tokens outside the decoded sequence, MOSS-VL widens its time-to-first-token advantage over same-backbone Qwen3-VL-8B from 2.8x to 5.1x as visual context grows. We release all five checkpoints, the training curriculum, and the real-time inference code at https://github.com/OpenMOSS/MOSS-VL.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a research paper introducing MOSS-VL, a new open vision-language model. The central claim is that watching and responding to live video — in real time — should be treated as a core design goal from the very beginning, not something bolted on afterward.
Alex: So the paper is asking why current video AI models have to finish watching a clip before they can say anything about it?
Sam: Exactly. Most open vision-language models process video the way you'd read a book — they take in the whole thing first, then respond. But in live settings, like security monitoring or a real-time assistant, a model needs to watch a scene unfold, decide on its own when something matters, and keep watching even while it's speaking.
Alex: And current models can't do that?
Sam: Not well. The problem is structural. Standard models feed visual information directly into the same processing pipeline as text. As the video gets longer, the computation required grows rapidly — and during the time the model is generating a reply, it effectively stops taking in new visual information. It goes blind mid-sentence.
Alex: So how does MOSS-VL get around that?
Sam: The key move is to separate visual data from the text generation process entirely. Think of it like this: in a standard model, every new video frame gets added to a long queue that the whole system has to re-read from the beginning. MOSS-VL instead stores incoming frames in a separate memory, and the language side of the model dips into that memory only when it needs to — without disrupting its own output. The technical name for this design is Gated Cross-Attention.
Alex: And that separation is what lets it keep watching while it talks?
Sam: Precisely. When a new frame arrives, it gets added to that memory cache without forcing the system to reprocess everything that came before. The result is that the model can start producing a response much faster, and that advantage grows the longer the video stream runs.
Alex: So that's the architecture. How does the model actually learn to behave the right way — to know when to speak and when to stay quiet?
Sam: That's where the training design comes in, and it's genuinely interesting. The model goes through a multi-stage training process. Early stages connect the visual and language components together, working with shorter sequences. Later stages open up the full model, expand how much context it can hold, and shift the training toward higher-quality material — things like mathematical reasoning and video clips where the timing of events matters.
Alex: But learning to handle a live stream seems like a different skill from just understanding video in general.
Sam: It is. That's handled in a final training stage the authors call Realtime-SFT. Here, training examples are constructed to mimic a live stream: video frames and text arrive in the order they would in real life, and after every single frame, the model has to make a decision.
Alex: What kind of decision?
Sam: Three options. Stay silent. Start speaking. Or, if it's already speaking, stop. It's a bit like teaching someone to be a good live commentator — knowing that most moments don't need a comment, but being ready to speak the instant something important happens.
Alex: That creates a problem, though. In most video streams, nothing interesting is happening most of the time. Wouldn't the model just learn to always stay quiet, because that's the safe answer?
Sam: That's exactly the risk, and the researchers took it seriously. Because silence vastly outnumbers speech in any real stream, a standard training setup would teach the model to default to silence and rarely bother speaking at all. To fix this, they applied a reweighting scheme that tracks how often each choice — speak or stay silent — appears across the training batch, and adjusts the importance of each accordingly. The rare moments when the model should speak get amplified so the system can't simply ignore them.
Alex: So it's like grading a student more harshly on the questions they got wrong most often, to make sure they actually learn those cases.
Sam: That's a fair way to put it. And that supervision approach, combined with carefully constructed training data for handling scene transitions and anchoring responses to specific events, is what enables the model to manage live streams without losing its ability to handle ordinary offline video tasks.
Alex: When you look at the evaluations, where does MOSS-VL actually hold up, and where does it fall short?
Sam: The picture is uneven, which is worth being clear about. The streaming advantages — particularly knowing when to speak — show up most clearly in tests that specifically measure that behavior. And the efficiency gains from the architectural design widen as visual history accumulates, which is exactly where you'd want them.
Alex: And on standard video understanding tasks?
Sam: The model holds its own on tasks that require reasoning about the timing and sequence of events in video. The core design does preserve offline capability while adding the real-time layer. But on reasoning-heavy exam benchmarks and document-focused tasks, it trails the strongest models at its scale — partly because it doesn't include a dedicated reasoning mode.
Alex: What about the specific behavior of adapting mid-reply — cutting off a response because something new just happened on screen?
Sam: That remains difficult to measure rigorously. The authors acknowledge that public streaming benchmarks don't yet test whether a model can interrupt itself when a scene changes unexpectedly. So that aspect of live behavior is real and designed for, but quantitatively underexplored.
Alex: Which points to where the field has to go next.
Sam: It does. The authors identify reinforcement learning as a likely next step for sharpening the model's decision-making, alongside the development of benchmarks that actually test live interactive behavior — not just offline understanding.
Alex: A clear piece of work with honest limits acknowledged. Thanks for listening to ResearchPod.