OpenMOSS Team
6 min
Abstract
We present MOSS-VL, an open vision-language model family that treats real-time interaction -- perceiving while it speaks -- as a first-class capability. It is co-designed across the stack: the language decoder attends to vision only through gated cross-attention, so the model can naturally see incoming frames while generating; a synthesized interaction corpus supervises when to speak, when to stay silent, and when to revise; and a staged curriculum concentrates all real-time-specific training in one light final stage over a strong offline foundation. Offline, MOSS-VL-Instruct is competitive at comparable scale and leads temporal-reasoning video sets. Across four streaming benchmarks, MOSS-VL-Realtime posts the best average on three (second on the fourth) among open-source streaming models, sweeping the three subsets that squarely test proactive behavior -- 66.0 vs. 37.5 for the best baseline on OmniMMI Proactive Alerting. With 11.3B parameters but visual tokens outside the decoded sequence, MOSS-VL widens its time-to-first-token advantage over same-backbone Qwen3-VL-8B from 2.8x to 5.1x as visual context grows. We release all five checkpoints, the training curriculum, and the real-time inference code at https://github.com/OpenMOSS/MOSS-VL.
Sam: It is. That's handled in a final training stage the authors call Realtime-SFT. Here, training examples are constructed to mimic a live stream: video frames and text arrive in the order they would in real life, and after every single frame, the model has to make a decision.
Alex: What kind of decision?
Sam: Three options. Stay silent. Start speaking. Or, if it's already speaking, stop. It's a bit like teaching someone to be a good live commentator — knowing that most moments don't need a comment, but being ready to speak the instant something important happens.
Alex: That creates a problem, though. In most video streams, nothing interesting is happening most of the time. Wouldn't the model just learn to always stay quiet, because that's the safe answer?
Sam: That's exactly the risk, and the researchers took it seriously. Because silence vastly outnumbers speech in any real stream, a standard training setup would teach the model to default to silence and rarely bother speaking at all. To fix this, they applied a reweighting scheme that tracks how often each choice — speak or stay silent — appears across the training batch, and adjusts the importance of each accordingly. The rare moments when the model should speak get amplified so the system can't simply ignore them.
Alex: So it's like grading a student more harshly on the questions they got wrong most often, to make sure they actually learn those cases.
Sam: That's a fair way to put it. And that supervision approach, combined with carefully constructed training data for handling scene transitions and anchoring responses to specific events, is what enables the model to manage live streams without losing its ability to handle ordinary offline video tasks.
Alex: When you look at the evaluations, where does MOSS-VL actually hold up, and where does it fall short?
Sam: The picture is uneven, which is worth being clear about. The streaming advantages — particularly knowing when to speak — show up most clearly in tests that specifically measure that behavior. And the efficiency gains from the architectural design widen as visual history accumulates, which is exactly where you'd want them.
Alex: And on standard video understanding tasks?
Sam: The model holds its own on tasks that require reasoning about the timing and sequence of events in video. The core design does preserve offline capability while adding the real-time layer. But on reasoning-heavy exam benchmarks and document-focused tasks, it trails the strongest models at its scale — partly because it doesn't include a dedicated reasoning mode.
Alex: What about the specific behavior of adapting mid-reply — cutting off a response because something new just happened on screen?
Sam: That remains difficult to measure rigorously. The authors acknowledge that public streaming benchmarks don't yet test whether a model can interrupt itself when a scene changes unexpectedly. So that aspect of live behavior is real and designed for, but quantitatively underexplored.
Alex: Which points to where the field has to go next.
Sam: It does. The authors identify reinforcement learning as a likely next step for sharpening the model's decision-making, alongside the development of benchmarks that actually test live interactive behavior — not just offline understanding.
Alex: A clear piece of work with honest limits acknowledged. Thanks for listening to ResearchPod.