Glonce

Researchers find on-policy distillation barely uses its teacher, propose OPSA instead

researchThe Glonce editorsHugging Face🔥56Glonce rating56Impact65Novelty70Relevance90Surprise70Players25Credibility80
Researchers find on-policy distillation barely uses its teacher, propose OPSA instead

A paper examines on-policy distillation (OPD), a technique that trains a student language model using dense, token-level scores from a teacher model, positioned as an alternative to the sparser reward signals used in reinforcement learning with verifiable rewards (RLVR). The authors point out a structural problem with OPD: the teacher is scoring trajectories generated by the student, which are inherently off-policy from the teacher's own perspective, so it is unclear how reliable that supervision actually is or whether it is really the source of the student's improvement. Quantitatively analyzing the teacher supervision produced during OPD training, the authors find substantial noise, and the amount of that noise grows as the teacher model gets larger. The surprising part is what happens next: the student policy turns out to be insensitive to this noise, converging to comparable performance whether the noisy supervision is kept or thrown out entirely. Digging into what is actually driving OPD's gains, the authors find that learning concentrates on tokens with low log-probability, and that replacing the teacher's per-token advantages with a single fixed negative advantage matches OPD's performance. The authors conclude that OPD works largely by suppressing low-probability tokens, a mechanism that does not require a teacher at all. Building on that finding, they propose On-Policy Self-Adaptation (OPSA), a supervision-free training method that uses entropy-adaptive negative advantages: it gives stronger learning signals to high-entropy positions, suppresses tail tokens, and redistributes probability mass evenly among head tokens. Tested against the base Qwen3-1.7B model, OPSA improves Avg@32 on the AIME24 math benchmark by 35.41 points, a 263% relative gain, and more than doubles Pass@32 across all three benchmarks used in the evaluation. Against OPD directly, OPSA comes out 16.77 points ahead in Avg@32 on AIME24. The authors report further experiments across other model families and tasks supporting the method's effectiveness and generalizability.

Key facts

  • On-policy distillation's teacher supervision contains substantial noise, and the noise increases with teacher scale.
  • The student model converges to comparable performance whether that noisy teacher supervision is kept or removed entirely.
  • OPD's actual gains trace to suppressing low log-probability tokens; a single fixed negative advantage matches teacher-provided advantages.
  • OPSA, a supervision-free method using entropy-adaptive negative advantages, needs no teacher model at all.
  • On Qwen3-1.7B, OPSA improves Avg@32 on AIME24 by 35.41 points (263% relative gain) over the base model and by 16.77 points over OPD.

Why it matters

On-policy distillation has been treated as a teacher-dependent technique: a larger, stronger model supervises a smaller student token by token. This paper argues that the teacher's contribution is mostly noise the student ignores, and that the real mechanism, suppressing low-probability tokens, works fine without any teacher. That reframes a class of distillation methods as something closer to a self-improvement trick than genuine knowledge transfer, and it directly motivates a simpler, cheaper method (OPSA) that removes the teacher from the loop.

Who it affects

Teams training or fine-tuning smaller language models for reasoning tasks, particularly anyone currently relying on a larger teacher model to run on-policy distillation. It also affects researchers studying RLVR and distillation methods, since the findings question a mechanism that underlies existing OPD pipelines.

How to use it

OPSA is described as a supervision-free method: it replaces teacher-provided per-token advantages with entropy-adaptive negative advantages computed from the student's own output distribution, giving stronger signals at high-entropy positions and suppressing tail tokens. That removes the need to host or query a separate teacher model during training, which is the main practical cost OPD carries.

How solid is it

The results are demonstrated primarily on Qwen3-1.7B, with AIME24 as the headline benchmark; the paper reports the method also more than doubles Pass@32 across all three benchmarks tested and states that further experiments across other model families and tasks support its generalizability, though the abstract does not name those additional benchmarks or model families.

Risks and caveats

The abstract gives only point differences and one relative-gain figure; it does not state the absolute Avg@32 or Pass@32 scores for the base model, OPD, or OPSA, nor does it identify the teacher model(s) used in the OPD comparison or their scale. The findings on teacher noise and student insensitivity come from the authors' own analysis and have not been independently verified here.