Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.
Alex: Welcome to another episode of ResearchPod. Today we're looking at VIBEVOICE-ASR-STREAMING, a technical report from Microsoft Research on real-time speaker-attributed transcription.
Sam: So the central question is whether you can collapse the traditional ASR-plus-diarization pipeline into a single, low-latency autoregressive pass?
Alex: Exactly. The problem with current unified models is that they're built for offline batch processing. They wait for the full audio, run a clustering algorithm over speaker embeddings, and then assign labels. That's fine for transcribing a recorded meeting after the fact, but it's completely incompatible with anything that needs to respond in real time.
Sam: And the clustering step is the real bottleneck—you can't assign speaker identities until you've seen enough audio to separate the embedding clusters, which typically means waiting until the conversation is nearly over.
Alex: Right. The authors report that cloud-based diarization APIs typically introduce around nine seconds of speaker-attribution latency. That's the gap this work is trying to close.
Sam: So what's the architectural move that makes real-time attribution possible?
Alex: The key insight is to treat speaker identity not as a post-hoc clustering problem, but as part of the token sequence itself. Rather than running a separate diarization module downstream, the model emits speaker labels interleaved with the transcription tokens—all in a single autoregressive pass. The context window is doing the work that a clustering algorithm would otherwise do. The model learns to track speaker identity as a property of the sequence, not as a geometric separation problem in embedding space.
Sam: So the model is essentially maintaining a running register of who's speaking, updated continuously as new audio arrives—and it doesn't need an external speaker embedding store at all.
Alex: That's exactly right. And because all of that history lives in the context window, the model can resolve speaker identity using both acoustic cues and lexical context—vocabulary, speaking style, topic continuity. That's a genuine architectural advantage over systems that treat diarization as a purely acoustic problem.
How does the chunking work in practice? That's where the latency actually gets set.
Alex: The model processes audio in fixed chunks of about 2.9 seconds, with a half-second lookahead buffer at each boundary. That lookahead gives the model a small window of future acoustic evidence before it commits to a speaker label—smoothing out the ambiguity that tends to cluster around turn boundaries.
Sam: But that lookahead is also a floor on your latency. You can't emit a label until you've seen those extra 500 milliseconds.
Alex: Exactly, and the authors are transparent about that trade-off. The reported expected speaker-attribution latency lands at around 2.0 seconds end-to-end. That's a meaningful reduction from the nine-second baseline, but it's not zero, and the lookahead is a direct contributor.
Sam: What does the accuracy picture look like? Is there a real cost to doing this in one pass rather than two?
Alex: That's the load-bearing empirical question, and the paper's answer is that the cost is modest and often absent. Across their evaluations, the streaming model achieves competitive word-error rates and diarization error rates relative to offline unified systems—and in some conditions matches or exceeds them. The authors attribute that partly to the lexical context the LLM brings to bear. The model can use the fact that one speaker consistently uses technical jargon, or that another always asks questions, to resolve segments that are acoustically ambiguous.
Sam: Where would a careful reviewer push back?
Alex: On the boundary conditions, mainly. The evaluation is solid but not exhaustive on adversarial conditions—heavy background noise, overlapping speech, or conversations with more than a handful of speakers. Those are the regimes where the acoustic signal degrades fastest and where you'd most want to know whether the lexical context advantage holds up.
Sam: And are there structural limits to the approach itself?
Alex: Two worth flagging. First, the context window has finite capacity, so very long conversations will eventually require some form of compression or truncation of the speaker history. The paper doesn't stress-test that regime. Second, the model's ability to track a given speaker depends on having seen enough of their speech to build a reliable in-context representation. For speakers who contribute very little audio early on, attribution accuracy is likely softer—though the paper doesn't break that out explicitly.
Sam: So the headline result is real, but the boundary conditions matter.
Alex: That's a fair read. The core contribution is demonstrating that an interleaved, autoregressive architecture can handle speaker attribution as part of the transcription task itself—no separate clustering stage—at latencies plausibly compatible with real-time applications. Whether that holds at scale, across longer conversations, noisier environments, and larger speaker sets, is the open question the field will need to answer next. Thanks for listening to ResearchPod.