Glonce

Qwen3, Llama and Gemma disagree on stop tokens, inflating distillation length

researchThe Glonce editorsHugging Face🔥52Glonce rating52Impact48Novelty68Relevance92Surprise58Players38Credibility82
Qwen3, Llama and Gemma disagree on stop tokens, inflating distillation length

On-policy distillation (OPD) trains a smaller student model to imitate a larger, post-trained teacher model's own generation process. The authors studied a failure mode of OPD in which student responses become excessively long, in some cases running until the generation budget is exhausted.

They trace this length inflation to a termination-token mismatch between the base student model and the post-trained teacher model. Testing across three model families, Qwen3, Llama and Gemma, they found that student and teacher can place their stopping probability on different end-of-sequence (EOS) tokens, even when the two models' declared sets of stop tokens are identical. This mismatch can suppress the student's own preferred way of ending a response without reliably transferring the teacher's preferred alternative in its place.

The authors also tested a direct fix: aligning the decoding stop-token sets between student and teacher. They report that this alone is not sufficient. What does substantially reduce the mismatch-driven length inflation, across all three model families, is treating functionally equivalent EOS tokens as a single shared semantic stopping action rather than as distinct tokens.

To see how this behavior develops over training, the authors also studied OPD across different stages of a training regime they name K2-Horizon; the text does not explain what K2-Horizon itself is beyond naming it as the training stages studied. Termination preferences shift substantially as training progresses, and a separate, distinct length-inflation effect appears late in the OPD run, one that persists even after the termination-alignment fix is applied.

The authors conclude that termination mismatch is an important but not exhaustive source of the length dynamics seen in OPD, meaning factors beyond EOS-token mismatch are also at play. They have released an implementation of the termination-handling corrections described in the paper.

Key facts

  • Length inflation in on-policy distillation (OPD) can make student responses run so long they exhaust the generation budget.
  • Across Qwen3, Llama and Gemma, student and teacher models can favor different EOS tokens even with identical declared stop-token sets, a termination mismatch that suppresses the student's own preferred stopping action.
  • Aligning stop-token sets alone does not fix the problem; treating functionally equivalent EOS tokens as one shared stopping action substantially reduces the inflation across all three model families.
  • Termination preferences shift over the course of training (studied across stages of a regime the authors call K2-Horizon), and a separate length-inflation effect appears late in training that persists after the alignment fix.
  • The authors call termination mismatch an important but not exhaustive cause of OPD length dynamics, and released an implementation of their corrections.

Why it matters

On-policy distillation is a common way to transfer a large, post-trained teacher model's behavior into a smaller student model. If the technique reliably makes outputs run longer than intended, and can exhaust the generation budget outright, that is a hidden cost anyone relying on distillation absorbs, in wasted compute and degraded output. Pinning the cause to a token-level termination mismatch, rather than to a training-data or capacity gap, changes where practitioners should look for a fix.

Who it affects

Anyone training a smaller model against a larger teacher through on-policy distillation, whether using Qwen3, Llama or Gemma, the three families studied here, or a similar architecture. Researchers building OPD pipelines, and engineers who inherit the resulting length or latency budgets, are the direct audience.

How to use it

The authors release an implementation of the termination-handling corrections they describe, built around treating functionally equivalent EOS tokens as one shared semantic stopping action rather than only aligning the two models' declared stop-token sets. The text does not describe the release's contents beyond that, and names no license, repository or usage instructions.

How solid is it

The central finding is tested empirically across three separate model families, Qwen3, Llama and Gemma, and further probed across multiple stages of a named training regime, K2-Horizon (which the text does not otherwise explain). That is a reasonably broad basis for the claim. The text gives no numeric magnitude for the length inflation itself or for how much the proposed correction reduces it, so the size of the effect cannot be assessed from the abstract alone.

Risks and caveats

The authors themselves describe termination mismatch as important but not exhaustive: a separate length-inflation effect shows up late in training and persists even after the termination-alignment fix is applied, so the correction proposed here does not fully close the gap. No author names or institutional affiliations appear in the text of the abstract, and the released implementation's contents are not described beyond termination-handling corrections.