Glonce

NVIDIA releases Nemotron 3.5 Transcribe and Granary v2 speech data

modelsThe Glonce editorsHugging Face🔥72Glonce rating72Impact65Novelty65Relevance100Surprise40Players85Credibility85
NVIDIA releases Nemotron 3.5 Transcribe and Granary v2 speech data

NVIDIA has published Nemotron 3.5 Transcribe, an open-weight, promptable model that turns recorded speech into text. It is available on Hugging Face and ModelScope. One checkpoint covers transcription, speaker attribution (diarization), translation and audio question answering. It transcribes 40 languages, translates 31 of them to or from English, and attributes speakers in 13. It handles recordings up to an hour long and labels up to eight speakers. Word-level timestamps are supported. Output is steered with plain instructions such as "Transcribe this recording and label each speaker," "Translate this audio into English," or "What decisions were made in this meeting?" Language support differs by task, and audio question answering is trained in English. For unfamiliar vocabulary, the post suggests a prompt that lists possible terms (for example HbA1c, metformin) and asks the model to use them only when the audio supports it.

Under the hood, the model pairs a speech encoder with a hybrid Mamba-Transformer mixture-of-experts language model, so only about 4B of its 30B parameters are active per token. Sortformer speaker cues are fed to the language model alongside the acoustic features, so each turn is tagged during generation rather than by a later step that attaches speakers to words.

The post leans on leaderboard results. On the Hugging Face Open ASR Leaderboard it is first among open-weight models for English at 5.33% average word error rate (WER), with API models excluded. In the model card's Open ASR comparison it also has the lowest WER among evaluated open-weight models on the multilingual track (4.62% mean across seven languages) and the long-form track (10.98% mean). On the Artificial Analysis Speech-to-Text Leaderboard, which ranks models independently by AA-WER, it ties for first among open-weight models at 2.8%, with about 4B active parameters versus 24B for the model it ties. On AA-AgentTalk, a held-out voice-agent speech set that makes up 50% of the index, it scores 1.8% WER against 2.3% for the next-best open-weight model. On Artificial Analysis's all-model chart it ranks ahead of several widely used proprietary speech APIs, which the post does not name.

For speaker attribution, David AI's DAI-ASR-I18N track for combined transcription and speaker labels puts it first among evaluated open-weight models and second overall. On that benchmark's diarization track across 21 languages it has the lowest diarization error rate of 14 systems, 18.26%, just ahead of NVIDIA's dedicated Nemotron 3 Diarization at 18.38%. On VoiceArena's Diarization-Bench (English full sessions, overlapping speech, 2 to 8 speakers) it posts the lowest cpWER of 14 systems, 32.4%, followed by NVIDIA Multitalker Parakeet with Nemotron 3 Diarization at 34.2%. It ranks first in both in-person and online settings there, and its deletions are 8.32% of reference words versus a median of about 31%. Ocular AI's Converse-STT evaluation ranks it the top open model, and Josh Talks' Voice of India leaderboard reports its results across Indian languages.

On speed, NVIDIA says the model processed 100 hours of audio in under three minutes on one NVIDIA RTX PRO 6000 Blackwell Server Edition GPU, using the forthcoming NVFP4 quantized version, measured on 30-second clips at high concurrency. The available checkpoint can be served with vLLM today. Quantized versions and local deployment with NeMo-Speech.cpp are "coming soon," aimed at RTX Spark, DGX Spark and Jetson AGX Thor as well as Blackwell data-center GPUs. The model is built for recorded audio, not live streaming. Agora, a real-time voice and video platform, used the word-level timestamps in a classroom demo that shows a transcript alongside the replay of a recorded class.

The second half of the post covers adaptation. Granary v2 is an open speech dataset and reusable curation pipeline built with NVIDIA NeMo Curator, part of the Nemotron speech collection on Hugging Face and ModelScope. It prepared multilingual speech used in Transcribe's training, producing aligned transcript variants, task labels and quality metadata. The public manifest covers 463,062 hours across 48 languages and 16 source corpora, up from 25 languages in Granary v1; the post stresses that these figures describe the dataset release, not Transcribe's full training mixture or sampling weights. Each recording gets three transcript forms (tn_raw, itn_raw, itn_clean), matching the verbatim and clean styles users can request by prompt. The pipeline also produces short audio captions, code-switching restoration of loanwords, speech-grounded Q&A pairs, contextual ASR biasing terms, and translations with automatic quality scores.

The pipeline has four stages. Audio preparation segments audio, tracks speakers and tags language and acoustic events; it was skipped for this release because the sources were already segmented. Transcription routes each language to primary and recovery ASR models and falls back when the primary fails a hallucination check, with human reference transcripts helping preserve trusted spellings. Task label generation derives transcript variants and checks character rate, hallucination patterns and agreement with models or human references; only accepted records get biasing terms and translations. The High Fidelity stage filters out noisy, clipped or multi-speaker segments to keep a smaller high-quality set; for VoxPopuli it cut 354k hours to 111k hours across 22 European languages. Developers can run the pipeline with Ray and NVIDIA Dynamo backends on their own recordings or authorized public corpora, or build on the public manifests, which also carry entity hints, derived instruction pairs, provenance and a per-record quality flag. Source audio must be downloaded separately under each corpus's original license.

A medical case study shows the progression from prompting to fine-tuning, using Eka Care's open benchmark of recordings from Indian doctors and medical students (3,619 clips, no designated training split). Naming English and the Latin alphabet in the prompt cut full-benchmark WER from 30.8% to 23.8%. Fine-tuning with NeMo Speech on roughly 153 hours of synthetic speech, 29 hours of accented English from Common Voice and 63 hours of medical recordings from MultiMed brought the full test set to 11.8% WER, about 62% fewer errors than the model out of the box. On held-out speakers, fine-tuning cut WER by 45.8% relative to the best prompt.

Key facts

  • Nemotron 3.5 Transcribe is an open-weight, promptable model on Hugging Face and ModelScope: it transcribes 40 languages, translates 31 to or from English, attributes speakers in 13, and handles recordings up to an hour with up to eight speakers.
  • It uses about 4B of its 30B parameters per token; it is first among open-weight models on the Open ASR English track at 5.33% average WER and ties for first on Artificial Analysis at 2.8% AA-WER.
  • On speaker attribution it posts the lowest DER on David AI's 21-language leaderboard (18.26% vs 18.38% for Nemotron 3 Diarization) and the lowest cpWER on VoiceArena (32.4% vs 34.2%).
  • Granary v2 covers 463,062 hours across 48 languages and 16 corpora, with a reusable NeMo Curator pipeline; source audio is downloaded separately under each corpus's original license.
  • In a medical case study on Eka Care's benchmark, a better prompt cut WER from 30.8% to 23.8% and fine-tuning reached 11.8%; the model is built for recorded audio, not live streaming.

Why it matters

Most open speech models return a transcript and stop. This one is meant to answer the follow-up questions in the same pass: who spoke, what it means in English, what was decided. It does that with natural-language instructions rather than separate models. NVIDIA also pairs the weights with the data pipeline and fine-tuning recipes behind them, so developers can see how the training targets were built and adapt the model to their own vocabulary. On Artificial Analysis, the post says it matches the top open-weight accuracy with about 4B active parameters versus 24B for the model it ties.

Who it affects

Developers building meeting transcription, call analysis, voice-agent pipelines, captioning and domain dictation such as medical notes. Teams working across many languages get 40 transcription languages in one checkpoint. Teams that need to adapt recognition to jargon or accents get the Granary v2 manifests, the NeMo Curator pipeline and NeMo Speech fine-tuning. Agora's classroom demo, which pairs a recorded class with a timestamped transcript, is one example from the post.

How to use it

The checkpoint is on Hugging Face and ModelScope and can be served with vLLM today. Instruct it in plain language, for example to label speakers, translate, or answer a question about the audio; the model card has recommended presets (including reasoning for audio questions), a full prompt guide and a task-by-language matrix. For domain terms, try a prompt first: list possible terms, say to use them only when the audio supports it, and specify language, script and transcript format. If errors remain, use the NeMo Speech fine-tuning workflow, which pairs representative speech with explicit written targets. Granary v2 manifests and the NeMo Curator pipeline (Ray and NVIDIA Dynamo backends) are available for building your own data. Quantized versions and local deployment with NeMo-Speech.cpp are announced as coming soon. The post points to the release's published terms for licensing.

How solid is it

This is a first-party announcement, so the framing is NVIDIA's. The benchmark claims point to third-party leaderboards: Hugging Face Open ASR, Artificial Analysis (which ranks independently), David AI and VoiceArena. Read the scope carefully. The Open ASR English first place is among open-weight models only, with API models excluded. The multilingual and long-form wins are within the model card's own comparison. The 100-hours-in-under-three-minutes figure is for the forthcoming NVFP4 quantized version on 30-second clips at high concurrency, not for the checkpoint available now. The lead over Nemotron 3 Diarization on David AI is narrow (18.26% vs 18.38%). The Granary v2 figures describe the dataset release, not Transcribe's full training mixture.

Risks and caveats

The model is built for recorded audio, not live streaming. Language support differs by task: 40 languages for transcription, 31 for translation to or from English, 13 for speaker attribution, and audio question answering is trained in English. The post itself advises spot-checking names, speaker changes and answers against the recording. Quantized versions and NeMo-Speech.cpp are not released yet. Granary v2 gives manifests, not audio: source audio must be downloaded separately under each corpus's original license. The medical case study covers one benchmark of Indian doctors and medical students, so the gains from prompting and fine-tuning there may not carry over to other domains.