Glonce

New method blends distillation and verifiable rewards for LLM post-training

researchThe Glonce editorsHugging Face🔥71Glonce rating71Impact70Novelty70Relevance95Surprise40Players35Credibility85
New method blends distillation and verifiable rewards for LLM post-training

A team of researchers has proposed On-policy Distillation with Verifiable Reward (OPDVR), a post-training method for large language models that merges two paradigms that have so far mostly been used separately: Reinforcement Learning with Verifiable Rewards (RLVR) and on-policy distillation (OPD). The authors note that each has a distinct weakness. RLVR only gives sparse, task-level feedback: a model finds out whether an entire answer was right or wrong, but gets no signal about which individual steps helped or hurt. OPD does the opposite: it gives dense, token-level guidance by having a student model match a teacher model's output distribution, but it ignores whether the resulting reasoning trajectory is actually correct, which caps the student's performance at roughly the teacher's own level. Prior attempts to combine the two approaches relied on weighted combinations or heuristic switching between them, which the authors say introduces extra hyperparameters and trade-offs that have to be tuned by hand. OPDVR avoids that by reformulating the implicit reward used in sampled-token OPD so that it is based on whether the sampled trajectory was correct, rather than purely on matching the teacher. It then applies a ReLU gating mechanism: correct trajectories are given non-negative rewards, and incorrect ones are given non-positive rewards. This keeps the distillation signal aligned with whether the task was actually solved, while still preserving the teacher's distributional guidance where it helps. A further effect of this reformulation is that it turns sampled-token OPD into a proper RLVR-style method, which the authors say makes it straightforward to plug into any policy gradient algorithm, including GRPO. In experiments across six reasoning benchmarks, the authors report that OPDVR consistently outperforms standard OPD. The paper does not name the six benchmarks or give the specific numeric margins by which OPDVR wins; it states only the qualitative result of consistent outperformance. The authors have released their code at github.com/LeapLabTHU/OPDVR.

Key facts

  • OPDVR combines on-policy distillation (OPD) with verifiable-reward reinforcement learning (RLVR) into one training method for large language models.
  • The method adds no extra hyperparameters, unlike prior approaches that combined the two via weighted mixing or heuristic switching.
  • It reformulates OPD's implicit reward around trajectory correctness and applies ReLU gating so correct trajectories get non-negative rewards and incorrect ones get non-positive rewards.
  • The reformulation turns sampled-token OPD into a proper RLVR method, making it compatible with policy gradient algorithms such as GRPO.
  • On six reasoning benchmarks, OPDVR is reported to consistently outperform standard OPD; code is public on GitHub.

Why it matters

Post-training large language models for reasoning typically leans on either RLVR or distillation, each with a known ceiling: RLVR's feedback is too coarse to reward good intermediate steps, and pure distillation cannot exceed the teacher model it is copying because it never checks whether the final answer was right. OPDVR's contribution is a way to get both a dense training signal and a correctness check at the same time, without the manual tuning that earlier hybrid approaches required.

Who it affects

The paper is aimed at researchers and engineers who build or fine-tune reasoning-focused language models, particularly teams already using either RLVR or on-policy distillation as a post-training step and looking for a way to combine the two without adding tuning overhead.

How to use it

The authors have published their implementation at github.com/LeapLabTHU/OPDVR. Because the method turns sampled-token OPD into a standard RLVR formulation, it is designed to drop into existing policy-gradient training pipelines, including ones already built around GRPO, rather than requiring a separate training framework.

How solid is it

The claim rests on experiments across six reasoning benchmarks, where the authors report OPDVR consistently beats standard OPD. The paper as summarized here does not specify which six benchmarks were used or give the numeric score gaps, so the strength of the result cannot be independently sized from this description alone; it also is not compared directly against RLVR alone or against other hybrid OPD/RLVR methods beyond general framing.

Risks and caveats

No benchmark names, numeric results, author affiliations, or publication date are given in the available text, which limits independent verification of the magnitude of improvement. The method's benefit is demonstrated specifically for reasoning tasks with verifiable answers; how it generalizes to tasks without a clear correctness signal is not addressed.