Glonce

Qwen3 study: on-policy delta distillation improves multilingual math reasoning

researchThe Glonce editorsHugging Face🔥76Glonce rating76Impact65Novelty60Relevance90Surprise40Players35Credibility80
Qwen3 study: on-policy delta distillation improves multilingual math reasoning

On-Policy Distillation (OPD) has been gaining attention as an alternative to reinforcement learning for post-training large language models, but researchers note that how well it works across languages had remained largely untested. A new paper studies OPD and a new variant called On-Policy Delta Distillation (OPD^2) specifically for mathematical reasoning in English, Korean and Japanese.

OPD^2 changes the training signal: instead of learning directly from a post-trained teacher model's output, it learns from the probability gap between that teacher and its own base model. The authors run experiments on Qwen3 and report that OPD^2 consistently outperforms the original OPD, with the strongest improvements showing up in Korean and Japanese, and that it generally narrows the performance gap between English and Korean.

The paper also tests a simpler setup: training OPD using only English data. That approach can still raise Korean and Japanese performance, but the authors find it often pulls the model's responses toward English even when the question was asked in another language. They conclude this points to the importance of including multilingual data in training, specifically to keep the model's answers in the language the user actually used.

The abstract does not give benchmark names, dataset details, or numeric scores for any of the comparisons, so while the direction of the results is stated plainly, their exact magnitude is not available from the text.

Key facts

  • On-Policy Distillation (OPD) is emerging as an alternative to reinforcement learning for LLM post-training, but its effectiveness across languages had been largely unexplored before this paper.
  • The paper introduces On-Policy Delta Distillation (OPD^2), which learns from the probability gap between a post-trained teacher model and its own base model, rather than from the teacher's output directly.
  • Tested on Qwen3 for math reasoning in English, Korean and Japanese, OPD^2 consistently outperforms plain OPD, with the strongest gains in Korean and Japanese, and it narrows the English-Korean performance gap.
  • Training OPD on English-only data also raises Korean and Japanese scores, but it tends to shift the model's responses toward English regardless of the question's language.
  • The authors conclude that multilingual training data is needed to keep responses in the target language rather than drifting to English.

Why it matters

On-Policy Distillation is being positioned as a lighter alternative to reinforcement learning for post-training language models, but until this paper its multilingual behavior had not been closely examined. Studying it directly for Korean and Japanese, not just English, matters because a post-training method that quietly favors English can leave non-English users with a weaker model even when the underlying architecture is identical.

Who it affects

Teams building or fine-tuning multilingual reasoning models, particularly for Korean and Japanese, and researchers evaluating on-policy distillation as an alternative to reinforcement learning for post-training. It is also relevant to anyone deploying Qwen3-based models for math tasks in these languages.

How to use it

OPD^2 is a training-time technique, not a released model or checkpoint. It works by using the probability gap between a post-trained teacher model and its own base model as the learning signal, rather than training directly on the teacher's output as plain OPD does. Reproducing it means implementing that delta-based signal during post-training; the abstract states no license, price, or code release.

How solid is it

The claims come from controlled experiments on the Qwen3 model family, comparing OPD^2 against plain OPD and against an English-only training variant across English, Korean and Japanese math reasoning. The authors state the improvement from OPD^2 is consistent and strongest in Korean and Japanese, and that it narrows the English-Korean gap, but the abstract gives no benchmark name, dataset, or numeric scores, so the actual size of the gains cannot be verified from the text alone.

Risks and caveats

The results so far cover one model family, Qwen3, and one task type, math reasoning, so how well they generalize elsewhere is untested here. The paper also flags a real limitation: training OPD on English-only data does lift Korean and Japanese performance, but it tends to make the model reply in English regardless of the language it was asked in, which undercuts the point of a multilingual system. The authors treat this as evidence that multilingual training data is necessary, not optional, for preserving target-language responses.

“OPD^2 improves OPD by using the probability gap between a post-trained teacher and its base model as the learning signal.”

— the authors, in the paper's abstract