Glonce

VibeVoice-ASR-Streaming adds real-time speaker tagging to speech recognition

researchThe Glonce editorsHugging Face🔥62Glonce rating62Impact55Novelty65Relevance85Surprise40Players20Credibility80
VibeVoice-ASR-Streaming adds real-time speaker tagging to speech recognition

Speaker-attributed ASR, transcribing speech while also labeling who said each word, has traditionally split into two separate stages: recognition and diarization. Models such as VibeVoice-ASR later merged both into a single system, but those unified models still mainly work offline, processing a full recording after the fact rather than as it streams in. That gap makes them a poor fit for real-time voice assistants and agents, which need low latency. A group of researchers now presents VibeVoice-ASR-Streaming, described as one of the first LLM-based end-to-end approaches to streaming speaker-attributed ASR. The method interleaves fixed-size chunks of audio, a small amount of lookahead audio, and the text produced so far, letting the model output "who said what" continuously as speech comes in, without a separate diarization pass. On accuracy, the 7B version of the model achieves the lowest average word and character error rate (WER/CER) across five evaluation sets. On speaker attribution, it comes out best or tied for best on 12 of 13 evaluation settings. The authors release both a 1.5B and a 7B version of the model weights, along with inference code.

Key facts

  • VibeVoice-ASR-Streaming performs speaker-attributed speech recognition in a streaming, low-latency setting rather than offline.
  • It works by interleaving fixed-size audio chunks, a small amount of lookahead audio, and previously produced text.
  • The 7B model achieves the lowest average WER/CER across five evaluation sets.
  • On speaker attribution, the model is best or tied-best on 12 of 13 evaluation settings.
  • The team releases 1.5B and 7B model weights together with inference code.

Why it matters

Combining transcription and speaker diarization into one model was already a step forward over running them as separate stages, but earlier unified models like VibeVoice-ASR worked mainly offline. That offline requirement blocks the low-latency use cases where speaker-attributed transcription is most wanted: live voice assistants and agents that need to know who is talking as the conversation happens, not after it ends. VibeVoice-ASR-Streaming targets exactly that gap by producing "who said what" incrementally as audio streams in.

Who it affects

The release is aimed at developers building real-time voice assistants and multi-speaker agent systems that need live transcription with speaker labels, such as meeting tools, call-center systems, or conversational agents that must track multiple participants as they speak.

How to use it

The authors release both a 1.5B and a 7B version of the model weights along with inference code, so the models can be run directly rather than only read about. No pricing, license terms, or release date are stated in the available material.

How solid is it

The reported results come from the authors' own evaluations: the 7B model posts the lowest average WER/CER across five evaluation sets, and it is best or tied-best on speaker attribution in 12 of 13 evaluation settings. The specific datasets, evaluation-set names, and comparison baselines behind those numbers are not given in the abstract, so the figures should be read as the authors' own reported benchmarks pending independent verification.

Risks and caveats

The abstract does not specify the size of the audio chunks or how much lookahead audio the streaming scheme uses, both of which directly affect the latency and accuracy tradeoff in practice. It also gives no comparison baselines or dataset details, so it is not possible from the available text to judge how large the gap is to prior streaming or offline systems, or how the approach performs outside the tested evaluation settings.