Glonce

SAF fixes entropy collapse in RLVR-distillation fusion for LLM training

researchThe Glonce editorsHugging Face🔥68Glonce rating68Impact55Novelty75Relevance98Surprise40Players35Credibility80
SAF fixes entropy collapse in RLVR-distillation fusion for LLM training

Researchers propose SAF (Stable Advantage Fusion), a training framework for large language models that combines two post-training reward signals: reinforcement learning with verifiable rewards (RLVR) and on-policy distillation (OPD). RLVR gives every token in a response the same reward, judged only at the sequence level. OPD instead scores each token individually against a stronger teacher model, producing a denser training signal, but it caps the student's performance at the teacher's own quality and discourages the student from exploring strategies the teacher does not already use.

Because the two methods have complementary strengths, combining them looks appealing, but the authors find that fusing the RLVR and OPD advantages with a fixed coefficient triggers entropy collapse: the training signal degenerates and the student stops exploring. They trace the collapse to two miscalibrations. First, a magnitude mismatch: token-level OPD advantages can spike far beyond the bounded RLVR advantage and erase its signal entirely. Second, a temporal mismatch: sustained full-strength OPD keeps pulling the student toward the teacher throughout training, limiting the exploration needed to eventually surpass it.

SAF addresses both problems with a four-stage pipeline applied only to the OPD advantage: a sparsify-then-compress mechanism for magnitude control, paired with a warm-up-then-anneal mechanism for temporal control. Each of the four stages is independently switchable and adds negligible computational overhead.

The authors instantiate RLVR with GRPO and evaluate SAF across seven mathematical-reasoning and code-generation benchmarks, using Qwen3 models at 1.7B, 4B and 8B parameters. SAF avoids entropy collapse and consistently outperforms fixed-coefficient GRPO+OPD fusion, improving the aggregate score by 0.51 to 2.70% across all six model-domain settings, while training more stably overall.

Key facts

  • SAF is a four-stage framework, sparsify-then-compress for magnitude control and warm-up-then-anneal for temporal control, that fixes miscalibrations when fusing RLVR and on-policy-distillation (OPD) advantages.
  • Fixed-coefficient fusion of RLVR and OPD advantages triggers entropy collapse from two miscalibrations: a magnitude mismatch, where OPD advantages overwhelm the bounded RLVR signal, and a temporal mismatch, where sustained OPD limits exploration.
  • SAF is tested with GRPO-based RLVR on Qwen3-1.7B/4B/8B across seven mathematical-reasoning and code-generation benchmarks, covering six model-domain settings.
  • SAF improves the aggregate score by 0.51 to 2.70% over fixed-coefficient GRPO+OPD fusion across all six settings, while avoiding entropy collapse and training more stably.
  • Each of SAF's four stages can be switched on or off independently and adds negligible overhead.

Why it matters

RLVR and on-policy distillation are two standard post-training approaches for language models with complementary weaknesses: RLVR gives a sparse, sequence-level reward that does not cap the student's ceiling but ignores which tokens actually mattered, while OPD gives a dense, token-level reward from a stronger teacher that caps performance at teacher quality and discourages exploration beyond it. Naively fusing the two advantage signals with a fixed coefficient looks like an obvious next step, but the authors show it backfires: it triggers entropy collapse, where the student's policy converges too fast and stops exploring useful alternatives. SAF is a targeted fix for that specific failure mode, making the RLVR-plus-distillation combination usable rather than just conceptually appealing.

Who it affects

Teams training or fine-tuning open language models that combine RLVR with distillation-style post-training, particularly using the GRPO algorithm and models in the Qwen3 family, tested here at 1.7B, 4B and 8B parameters. It is also relevant to researchers benchmarking math-reasoning and code-generation performance of mid-size open models.

How to use it

SAF is described as a lightweight pipeline applied only to the OPD advantage, combining a sparsify-then-compress mechanism for magnitude control with a warm-up-then-anneal mechanism for temporal control. Each of the four stages can be switched on or off independently, and the framework adds negligible overhead on top of standard GRPO training, so it is presented as a drop-in addition to existing RLVR-plus-OPD pipelines rather than a separate training regime. The source does not mention a code or model release.

How solid is it

The claims rest on results across seven mathematical-reasoning and code-generation benchmarks and three Qwen3 model sizes, 1.7B, 4B and 8B, giving six model-domain settings in total. SAF is reported to consistently outperform fixed-coefficient GRPO+OPD fusion in all six, with the aggregate-score improvement ranging from 0.51 to 2.70%. The text gives only this relative-improvement range, not absolute benchmark scores, and does not state which of the four stages contributes most to the gain.

Risks and caveats

The reported gains are modest, 0.51 to 2.70% on the aggregate score, and are measured only against a fixed-coefficient fusion baseline rather than against other distillation approaches. The source does not name the paper's authors, their institution, or a publication venue, and it does not report absolute accuracy numbers on the seven benchmarks, only the relative improvement over that one baseline.