Glonce

Hugging Face adds Hindi and Indian English to the Open ASR Leaderboard

researchThe Glonce editorsHugging Face🔥69Glonce rating69Impact62Novelty65Relevance95Surprise42Players55Credibility82
Hugging Face adds Hindi and Indian English to the Open ASR Leaderboard

Hugging Face's Open ASR Leaderboard ranks speech recognition models by word error rate (WER), and that ranking shapes which models get adopted and improved, while capabilities the leaderboard does not measure tend to stagnate. Recent work on the leaderboard has focused on making that single WER number harder to game: held-out private splits, analysis to check whether models are reproducing reference transcripts rather than transcribing the audio, and closing gaps in the text normalisers so correct variant spellings are not penalised. None of that addresses a separate, longstanding problem: a study titled 'Racial disparities in automated speech recognition' found commercial ASR systems roughly twice as bad for Black speakers as for white speakers, and a second study, 'Quantifying Bias in Automatic Speech Recognition,' found further error-rate differences by gender, age and accent. A leaderboard built on test sets that record what was said and almost nothing about who said it cannot see any of this.

To close that gap, Hugging Face has added two evaluation sets, Monsoon en-IN and Monsoon hi-IN, each released as a public split for self-scoring plus a private split withheld to limit benchmark-specific optimisation. Hindi, spoken by more than half a billion people, becomes the first Indic language on a multilingual leaderboard tab that until now covered only European languages. The four splits are speaker-disjoint and comprise 4,888 speakers in total, with 12 attributes recorded for each speaker alongside the transcript, for 18 columns per segment overall, well beyond the identifier, transcript and duration that most public ASR test sets ship.

The sets were built to vary along nine axes: geography, age, gender, vocabulary, devices, acoustic environments, speech type, speech rate, and the existence of multiple valid transcripts for the same audio, on the reasoning that a test set can only expose a failure mode it varies along. Contributors were recruited through the Voice Arena community, described as a global digital platform whose reach extends into rural and semi-urban districts that speech corpora rarely cover, and recorded unscripted, dual-channel, peer-to-peer conversations on assigned everyday topics such as travel, healthcare, agriculture, education and digital services, using their own handsets and their own network connections rather than supplied hardware in a quiet room. Each conversation opened with a narrative cue and progressively revealed follow-up questions, with candidate topics generated by large language models and then reviewed and localised by native-speaker linguists. Contributors passed a language-proficiency screening before recording, were compensated, and gave informed consent covering use in training and distribution; a per-speaker duration cap, calibrated per language, stopped a small number of prolific contributors from dominating a language or region.

Quality control ran every recording through automated gates before transcription: language identification models trained across more than 30 languages verified the spoken language, a gender-confirmation classifier corroborated (but did not replace) the self-reported gender label, a separate model distinguished live spontaneous speech from pre-recorded or played-back audio, and signal-to-noise estimation removed unintelligible recordings while deliberately preserving natural background noise. Recordings were then segmented by voice-activity detection at two seconds of continuous silence or a fifteen-second soft cap, and passed a DNSMOS P.808 quality check. Transcription itself was human work under a five-level protocol: a first machine-generated draft, produced by internal ASR models that appear on no public leaderboard (so no system evaluated on Monsoon contributed to its own reference), was corrected and independently re-verified by different native-speaking linguists across multiple rounds, with numerals written out as words. For Hindi, where spelling variants are not a fixed mapping between two conventions, the reference ships as a lattice listing every spelling accepted as correct for each span of the transcript, rather than relying on a normaliser as the English sets do.

The resulting sets are deliberately spread thin rather than deep: the ten largest contributors account for only 2.8% to 6.8% of total recorded duration, and more than half of all 4,888 speakers appear in exactly one segment, so no single voice can carry a score. The Indian English public split draws on 428 native districts across 30 states and union territories, while the two Hindi splits span 202 and 295 districts; recordings come from 315 to 582 distinct device models, with no single model exceeding 2.1% of segments in any subset. The Indian English public set is geographically flat: all six zones are represented, with 35% of segments from southern speakers, 18% from the East, 18% from Central, 16% from the North and 11% from the West, and no single state exceeding 13% of speakers, a third of whom come from outside the eight largest states. The Hindi sets, being a Hindi-belt language, concentrate more tightly, with Uttar Pradesh accounting for roughly 40% of speakers, matching what a population-weighted Hindi sample should look like. Metadata recorded per clip includes age, gender, district, state, occupation, education, marital status, income band, handset brand, current city and years in the current district; five sample clips from the public Indian English split range from a 29-year-old student in West Tripura recording on a Samsung SM-G781B to a 57-year-old man in Puducherry recording on a Xiaomi M2006C3LI.

Hugging Face illustrates why the metadata matters with one worked example on the Indian English public split, framed as an illustration rather than the paper's main finding: eight models on the leaderboard land between 4.81 and 4.99 WER, a spread of only 0.18 points from best to worst, which the post says is inside what five hours of audio can resolve; ranked on the aggregate corpus alone, the eight models are effectively indistinguishable. Grouping the same speakers by region, using each speaker's native district rolled up to its Ministry of Home Affairs zonal council, is described as telling a different story, though the retrieved text cuts off before the specific per-zone numbers are given. The post separately cites an earlier closed-benchmark analysis of Indian ASR that found district-level error rates spanning roughly 4% to 44%, with underrepresented regions trailing the Hindi belt and major metros; Monsoon is presented as making that same class of region-level analysis possible on a public leaderboard test set for the first time.

Key facts

  • Hugging Face added two new evaluation sets, Monsoon en-IN and Monsoon hi-IN, making Hindi (spoken by more than half a billion people) the first Indic language on an Open ASR Leaderboard tab that previously covered only European languages.
  • The four speaker-disjoint splits comprise 4,888 speakers with 12 recorded attributes each, collected through the Voice Arena platform from unscripted conversations recorded across hundreds of Indian districts on contributors' own handsets.
  • No single voice, region or device dominates the data: the ten largest contributors account for just 2.8% to 6.8% of total duration, more than half of all speakers appear in exactly one segment, and no device model exceeds 2.1% of segments.
  • On the Indian English public split, eight leaderboard models score within 0.18 WER points of each other (4.81 to 4.99), which Hugging Face uses to illustrate how an aggregate score can look uniform while regional breakdowns, now possible thanks to the new speaker metadata, reveal differences the aggregate conceals.
  • The project responds to prior findings that commercial ASR systems are roughly twice as error-prone for Black speakers as for white speakers, and that further gaps exist by gender, age and accent, disparities a standard leaderboard test set cannot show because it records what was said but almost nothing about who said it.

Why it matters

The Open ASR Leaderboard's single WER number decides which speech models get adopted and iterated on, but a leaderboard test set that records only what was said, not who said it, cannot reveal that error rates fall unevenly across speakers. Prior research cited in the post found commercial ASR systems roughly twice as bad for Black speakers as for white speakers, plus further gaps by gender, age and accent, none of which shows up in an aggregate score. Adding Hindi and Indian English as the first Indic-language entries, built with per-speaker metadata from the start, is meant to make that kind of disparity visible and measurable rather than invisible by construction.

Who it affects

Teams that build, fine-tune or select ASR models using the Open ASR Leaderboard as a signal now have a benchmark that can be broken down by region, device, age and gender rather than only ranked in aggregate. It also matters to the roughly half a billion Hindi speakers and to Indian English speakers across all six of the country's linguistic zones, whose speech, dialects and recording conditions had no dedicated representation on the leaderboard before this addition.

How to use it

Monsoon en-IN and Monsoon hi-IN are each released with a public split, available now for self-scoring by anyone submitting a model to the leaderboard, and a private split withheld to limit benchmark-specific optimisation. The Hindi sets ship references as a lattice of accepted spelling variants for each transcript span rather than relying on a normaliser, since Hindi spelling variation is not a fixed mapping between conventions the way the English sets' standard string references allow.

How solid is it

Every recording passed automated gates before transcription: language identification trained across more than 30 languages, a gender-confirmation classifier used to corroborate rather than replace the self-reported label, a separate model to catch played-back rather than live audio, signal-to-noise filtering, voice-activity-detection segmentation, and a DNSMOS P.808 quality check. Transcripts were then produced by native-speaking linguists under a five-level protocol with independent verification at each stage, starting from a machine draft generated by internal ASR models that do not appear on any public leaderboard, so no model evaluated on Monsoon contributed to its own reference. Concentration checks (2.8% to 6.8% of duration from the ten largest contributors, over half of speakers appearing once, no device model above 2.1% of segments, 202 to 428 districts and 315 to 582 device models represented) support the claim that no single voice, region or device can carry the score.

Risks and caveats

The worked example in the post, eight leaderboard models scoring within 0.18 WER points of each other in aggregate, is offered only as an illustration of what regional breakdown can reveal, and the retrieved text cuts off before the specific per-zone results are given, so the actual size of the regional gap on this set is not established here. Coverage is also still narrow: only Hindi and Indian English are added, geographic representation differs by design between the flatter English sets and the more Hindi-belt-concentrated Hindi sets, and the post gives no timeline for when further Global South languages might follow.

“A test set can only expose a failure mode it varies along.”

— the Hugging Face blog post