Glonce

VākQA benchmark tests AI on Telugu spoken question answering

researchThe Glonce editorsHugging Face🔥69Glonce rating69Impact55Novelty78Relevance100Surprise48Players20Credibility82
VākQA benchmark tests AI on Telugu spoken question answering

Researchers led by Bhavana Akkiraju and co-authors introduced VākQA, a spoken question answering (SQA) benchmark for Telugu, a language that had no such benchmark before despite rapid progress on QA for high-resource languages. VākQA consists of 2,001 factoid question-answer pairs spanning six domains, built from 2.53 hours of speech audio, with bilingual transcriptions and human-verified reference answers. Before benchmarking any model, the authors first checked whether automatic evaluation itself could be trusted for Telugu: they compared several evaluation methods against human judgements and found that Gemini-as-a-judge came closest to matching human ratings, though its strictness was not applied uniformly across cases. Open-weight judge models performed worse in a specific way: they systematically marked down correct Telugu answers whenever the wording differed from the reference answer's surface form, even when the meaning was right. Using this human-validated evaluation setup, the team then benchmarked proprietary and open-weight models across different input modalities (text versus speech), languages, and domains. Three patterns emerged. Telugu phrasing carries cultural specificity that gets lost when questions or answers are translated. Feeding speech directly as input introduces phonetic confusions that can change what a question is actually asking. And in cascaded pipelines, where speech is first converted to text by automatic speech recognition (ASR) and then machine-translated (MT), errors from the two stages compound progressively rather than canceling out. The paper does not name the specific models tested or report numeric accuracy scores for them, nor does it give the authors' institutional affiliation or a release date. VākQA itself is publicly released.

Key facts

  • VākQA is a Telugu spoken factoid QA benchmark with 2,001 question-answer pairs across six domains, built on 2.53 hours of speech audio with bilingual transcriptions and human-verified reference answers.
  • Among automatic evaluation methods tested against human judgements, Gemini-as-a-judge came closest to human ratings but applied its strictness unevenly.
  • Open-weight judge models systematically penalized correct Telugu answers whenever their wording differed from the reference answer's surface form.
  • Feeding speech directly as input introduces phonetic confusions that can alter what a question means, and cascaded ASR-to-translation pipelines compound errors across stages.
  • Telugu phrasing carries cultural specificity that is lost when translated, and the benchmark is publicly released.

Why it matters

Spoken question answering has improved fast for high-resource languages, but Telugu, spoken by tens of millions of people, had no benchmark to measure it at all. VākQA fills that gap, and it does something benchmarks for major languages rarely bother to do: it checks whether the automatic scoring methods used to grade AI answers are themselves reliable for this language, before using them to grade anything.

Who it affects

Teams building or evaluating speech assistants, voice search, or QA systems for Telugu and other low-resource languages; researchers working on ASR-to-translation pipelines who need to know where errors enter and accumulate; and anyone relying on an LLM-as-judge to score non-English output, since the failure mode found here (penalizing correct answers phrased differently from the reference) is not specific to Telugu.

How to use it

VākQA is publicly released, so it can be used directly to benchmark spoken QA systems on Telugu and to sanity-check an evaluation judge on this language before trusting its scores. The source does not give a license, access process, or pricing, so none of that can be stated here.

How solid is it

The authors did not just benchmark models: they first validated their evaluation methodology against human judgements, which is what let them trust Gemini-as-a-judge's scores enough to use it for the model comparison that followed. That said, the available text does not name the specific proprietary and open-weight models tested, nor does it report the numeric accuracy or score results from that comparison, so the benchmark's existence and validated methodology are established, but not its concrete leaderboard results.

Risks and caveats

The clearest risk is in evaluation itself: open-weight judge models marked down answers that were correct but worded differently from the reference, which means teams using such judges on Telugu (or similarly under-resourced languages) risk under-scoring working systems. On the input side, cascaded ASR-then-translation pipelines were shown to compound errors progressively rather than average them out. The source does not name the authors' institution, give a release date, list the specific models benchmarked, or report numeric results, so those remain open questions.

“Telugu phrasing retains cultural specificity that is lost in translation, speech input introduces phonetic confusions that alter question meaning, and cascaded ASR-MT errors compound progressively.”

— VākQA paper