Yujie Tu, Zhiliang Peng, Jianwei Yu, Li Dong, Songchen Xu, Yaoyao Chang, Wenhui Wang, Zilong Wang, Zehua Wang, Yan Xia, Jiajun Zhang, Xie Chen, Furu Wei
5 min
Abstract
Traditional speaker-attributed ASR systems treated ASR and speaker diarization as two separate tasks. Recently, end-to-end models such as VibeVoice-ASR have unified the two tasks within a single model. However, existing unified models still mainly support offline recognition, making it difficult to meet the low-latency requirements of real-time voice assistants and agents. To tackle this issue, we present VibeVoice-ASR-Streaming, one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. It interleaves fixed-size audio chunks, a small amount of lookahead audio and previous text. This allows the model to produce ''who said what'' as speech arrives, without a separate diarization stage. For transcription accuracy, our 7B model achieves the lowest average WER/CER across five evaluation sets. For speaker attribution, it achieves the best or tied-best on 12 of 13 evaluation settings. We release the 1.5B and 7B model weights together with inference code.
Alex: Exactly, and the authors are transparent about that trade-off. The reported expected speaker-attribution latency lands at around 2.0 seconds end-to-end. That's a meaningful reduction from the nine-second baseline, but it's not zero, and the lookahead is a direct contributor.
Sam: What does the accuracy picture look like? Is there a real cost to doing this in one pass rather than two?
Alex: That's the load-bearing empirical question, and the paper's answer is that the cost is modest and often absent. Across their evaluations, the streaming model achieves competitive word-error rates and diarization error rates relative to offline unified systems—and in some conditions matches or exceeds them. The authors attribute that partly to the lexical context the LLM brings to bear. The model can use the fact that one speaker consistently uses technical jargon, or that another always asks questions, to resolve segments that are acoustically ambiguous.
Sam: Where would a careful reviewer push back?
Alex: On the boundary conditions, mainly. The evaluation is solid but not exhaustive on adversarial conditions—heavy background noise, overlapping speech, or conversations with more than a handful of speakers. Those are the regimes where the acoustic signal degrades fastest and where you'd most want to know whether the lexical context advantage holds up.
Sam: And are there structural limits to the approach itself?
Alex: Two worth flagging. First, the context window has finite capacity, so very long conversations will eventually require some form of compression or truncation of the speaker history. The paper doesn't stress-test that regime. Second, the model's ability to track a given speaker depends on having seen enough of their speech to build a reliable in-context representation. For speakers who contribute very little audio early on, attribution accuracy is likely softer—though the paper doesn't break that out explicitly.
Sam: So the headline result is real, but the boundary conditions matter.
Alex: That's a fair read. The core contribution is demonstrating that an interleaved, autoregressive architecture can handle speaker attribution as part of the transcription task itself—no separate clustering stage—at latencies plausibly compatible with real-time applications. Whether that holds at scale, across longer conversations, noisier environments, and larger speaker sets, is the open question the field will need to answer next. Thanks for listening to ResearchPod.