Glonce

Accuracy benchmarks miss whether frontier AI models reason in Greek at all

researchThe Glonce editorsHugging Face🔥84Glonce rating84Impact70Novelty78Relevance90Surprise72Players60Credibility82
Accuracy benchmarks miss whether frontier AI models reason in Greek at all

A research paper takes three frontier mixture-of-experts models, one each from Alibaba, OpenAI and NVIDIA, with 3.6 to 4.0 billion active parameters, and fine-tunes them to reason in Greek, chosen as a case study in low-resource-language reasoning. The training runs first through supervised fine-tuning (SFT), then through reinforcement learning with verifiable rewards (RL), and the authors measure the results both on accuracy benchmarks and on six behavioural dimensions they propose specifically because accuracy cannot capture them.

On the accuracy benchmarks, almost nothing happens, and the authors argue the benchmark itself is noise at this scale: changing only the random seed moves the score by 7.7 points, more than every data and training-recipe effect they measured. They report this null result first.

The real change is invisible to accuracy. Base, pre-fine-tuning models never reason in Greek: 0 of 1,000 reasoning traces are in Greek, even when the question itself is posed in Greek, so a model can answer correctly while reasoning in a form its user cannot read, audit or correct. After SFT, every released checkpoint reasons in the language of the question on about 98% of items, with one model family needing three times fewer tokens to do so. Judged grammaticality improves on all four fine-tuned models, and general ability stays within a few points of each base model, so the authors conclude nothing was forgotten while fluency was gained.

SFT does not fix everything, though. A quarter of answers skip the requested output format, some answers leak into the reasoning channel instead of the answer field, and an explicit instruction to "think in English" is obeyed under half the time. Reinforcement learning with verifiable rewards, pre-registered before training began, fixes the first two defects outright: the format-skipping rate drops from 24% to 2.5%, and the leak rate drops from 3.5% to 0.0%, both measured against a flat random-reward control. RL also moves compliance with the "think in English" instruction up by 9.1 percentage points, though no absolute post-RL compliance rate is given. The Greek-reasoning habit itself, however, survives an accuracy-only training gradient untouched: reinforcement learning that rewards only correctness does not remove it.

The authors release five checkpoints from the work. They also report six instrument failures of their own, each caught by a control they built in, and argue that the instruments, controls and pre-registration approach travel to any low-resource language; Greek was simply the case that let them measure it.

Key facts

  • Changing only the random seed shifts the accuracy-benchmark score by 7.7 points, more than any data or training-recipe effect the authors measured, making the benchmark itself noise at this scale.
  • Base, pre-fine-tuning models never reason in Greek: 0 of 1,000 reasoning traces were in Greek even when the question itself was posed in Greek.
  • After supervised fine-tuning, every released checkpoint reasons in the question's language on about 98% of items, with one model family using three times fewer tokens and general ability staying within a few points of its base model.
  • SFT alone leaves defects accuracy cannot see: a quarter of answers skip the requested format, some answers leak into the reasoning channel, and an explicit "think in English" instruction is obeyed under half the time.
  • Reinforcement learning with verifiable rewards cuts the format-skip rate from 24% to 2.5% and the leak rate from 3.5% to 0.0%, but the Greek-reasoning habit itself survives accuracy-only RL untouched.

Why it matters

Standard accuracy benchmarks can stay almost flat while a model's entire reasoning language changes underneath them. A model that answers correctly in a low-resource-language setting can still be reasoning in English internally, in a form its actual user cannot read, audit or correct. This paper argues that treating accuracy as a proxy for language-appropriate reasoning is a category error: the two barely correlate, and the gap only shows up once you build metrics designed to see past accuracy.

Who it affects

Teams building or evaluating multilingual and low-resource-language reasoning models, anyone relying on a model's visible reasoning trace as an audit or safety mechanism, and researchers designing reinforcement-learning pipelines with verifiable rewards who need to know which defects RL actually fixes and which it leaves untouched.

How to use it

The authors release five fine-tuned checkpoints from the study. They present the six behavioural dimensions, the controls that guard each one, and the pre-registration approach as reusable on any low-resource language, not just Greek. No pricing, licence terms or model names are given in the source, so none are stated here.

How solid is it

The behavioural metrics are gated specifically to reject anything that correlates with output length, and the RL comparisons run against a flat random-reward control rather than an untrained baseline, which rules out the reward signal itself as the explanation. The authors also self-report six instrument failures of their own and how each was caught, which reads as a real methodological safeguard rather than a polished result. Missing from the source are the specific model names or versions behind "Alibaba," "OpenAI" and "NVIDIA," the researchers' names and institution, the name of the accuracy benchmark used, and any date for the work or the checkpoint releases.

Risks and caveats

The claim of generalising beyond Greek to any low-resource language is the authors' own framing; only Greek was actually tested. The RL improvement on the "think in English" instruction is reported only as a 9.1 percentage point change, with no absolute post-RL compliance rate given. The source does not specify how grammaticality was judged, human raters or an automated judge, or exactly what the flat random-reward control condition consisted of.

“Base models never think in Greek: 0 of 1,000 reasoning traces, even when the question is Greek, so the model answers correctly while reasoning in a form its user cannot read, audit, or correct.”

— the paper's authors