On-policy distillation paper: training-induced differences transfer, scale-only ones barely do
On-policy distillation (OPD) trains a target policy on its own rollouts, using token-level supervision from a stronger teacher. The usual catch is that it relies on the teacher being substantially more capable than the target. Recent work relaxes that requirement by transferring the relative policy difference between an expert and a reference policy, instead of the expert's absolute distribution. What has been unclear is which differences give useful supervision and which targets can benefit from them. This paper sets out to answer both questions.
The authors build a unified view of the setup. In it, the log-probability ratio between a proxy expert and a proxy reference defines an implicit reward for KL-regularized policy optimization. That formulation induces an on-policy update for the target, recovers conventional OPD as a special case, and allows any-to-any transfer across models that differ in scale, training stage or generation.
Their systematic experiments yield three findings. First, a stronger expert is not necessarily a better source of supervision: differences induced by task-specific training, including reinforcement learning and supervised fine-tuning, transfer consistently, whereas those arising from model scale alone transfer weakly or not at all. Second, the expert and the reference play complementary roles: the expert provides the capability to be transferred, while the reference filters out shared behavior and isolates the relevant residual. Third, even a useful difference does not transfer to every target: success requires the induced update to move the target toward the expert and to generalize beyond sampled rollouts.
Two further observations round this out. Raw policy distance does not predict transferability, and an appropriate reference can rescue an otherwise unsuccessful transfer. Taken together, the authors say, the results characterize when relative policy differences provide effective supervision for OPD.
Key facts
- The paper frames on-policy distillation so that the log-probability ratio between a proxy expert and a proxy reference acts as an implicit reward for KL-regularized policy optimization.
- The framing recovers conventional OPD as a special case and allows any-to-any transfer across models that differ in scale, training stage or generation.
- Differences induced by task-specific training, including reinforcement learning and supervised fine-tuning, transfer consistently; differences from model scale alone transfer weakly or not at all.
- The expert supplies the capability to transfer, while the reference filters out shared behavior and isolates the relevant residual.
- Raw policy distance does not predict transferability, and an appropriate reference can rescue an otherwise unsuccessful transfer.
Why it matters
Conventional on-policy distillation leans on a teacher that is much more capable than the target. Transferring only the relative difference between an expert and a reference loosens that demand, but until now it was unclear which differences are worth transferring. This paper gives a structured answer: a stronger expert is not necessarily a better source of supervision, and what matters is how the difference between expert and reference was produced. It also offers a single formulation in which standard OPD is a special case.
Who it affects
Researchers and engineers who train a target policy with on-policy distillation, or who are considering transferring the difference between an expert and a reference policy. The findings bear on how they choose the expert, how they choose the reference, and which targets to expect results from.
How to use it
The abstract offers guidance on choosing the pieces of a transfer. Prefer expert-reference differences that come from task-specific training such as RL or SFT over differences that come from scale alone. Treat the reference as a filter that removes shared behavior, so pick it with care; an appropriate one can rescue a transfer that otherwise fails. Do not use raw policy distance as a predictor of success. The source mentions no code, release or availability.
How solid is it
Only the abstract is available. It reports three findings from systematic experiments, but gives no models, model sizes, benchmarks, datasets or quantitative results, and does not state which model pairs were used as expert, reference or target. It names no authors or institutions. The strength of the evidence cannot be judged from this text alone.
Risks and caveats
The authors themselves state that a useful difference does not transfer to every target: the induced update must move the target toward the expert and generalize beyond sampled rollouts. Differences arising from model scale alone transfer weakly or not at all, so a bigger expert is no guarantee of a gain. Because the abstract carries no numbers, the size of any effect is unknown.
“a stronger expert is not necessarily a better source of supervision”
— Paper abstract