Glonce

Researchers trace FP8 reinforcement learning instability to a clipping bug, fix it with Calibrated Clipping

researchThe Glonce editorsHugging Face🔥76Glonce rating76Impact70Novelty78Relevance100Surprise60Players25Credibility82
Researchers trace FP8 reinforcement learning instability to a clipping bug, fix it with Calibrated Clipping

Reinforcement learning (RL) has become a core technique for improving reasoning and agentic behavior in large language models, and running RL in FP8 precision can speed up training. But a new paper finds that keeping an FP8 RL pipeline stable end to end is harder than assumed. Prior work tried to fix mismatches between training and inference precision using correction techniques such as TIS, but the authors show that full-pipeline FP8 RL still suffers from severe instability: entropy suddenly surges partway through training, and models start producing garbled output. Tracing the cause, the authors find a previously overlooked problem: quantization noise from FP8 compounds over the pipeline and distorts the importance ratio used in RL updates. This distortion disproportionately pushes tokens with negative advantage outside the trust region, which erroneously zeroes out their gradients. Because those gradients are zeroed, pathological (bad) outputs are not properly penalized during training and instead accumulate, eventually producing the observed entropy spikes and garbled text. To fix this, the authors propose Calibrated Clipping, a dynamic method that aligns the FP8 clipping bounds with the distributions seen in high-precision BF16 training: it matches the lower-bound clipping quantile to the BF16 reference and rebalances the upper bound accordingly. In extensive experiments across two RL algorithms, GRPO and DAPO, model scales ranging from 8B to 32B parameters, and multiple FP8 scaling granularities, the method eliminates the entropy surges and restores performance comparable to the BF16 baseline. The paper does not name specific authors, an institution, a publication date, exact numeric performance figures for the FP8-versus-BF16 comparison, or the datasets and benchmarks used in the experiments.

Key facts

  • Full-pipeline FP8 reinforcement learning for LLMs suffers from severe instability: mid-training entropy surges and garbled outputs, even with prior correction techniques like TIS.
  • The root cause: compounded FP8 quantization noise distorts the importance ratio, pushing negative-advantage tokens outside the trust region and erroneously zeroing their gradients, so bad outputs go unpenalized and accumulate.
  • The proposed fix, Calibrated Clipping, dynamically aligns FP8 clipping bounds with high-precision BF16 distributions by matching the lower-bound quantile and rebalancing the upper bound.
  • Tested across GRPO and DAPO algorithms, model scales from 8B to 32B, and multiple FP8 scaling granularities, the method eliminates entropy surges and matches BF16 baseline performance.

Why it matters

FP8 quantization promises faster, cheaper RL training for large language models, but only if the pipeline stays numerically stable. This paper identifies a specific, previously unrecognized failure mode: quantization noise corrupting the importance ratio in a way that silently disables gradient penalties on bad outputs, letting them accumulate until training visibly breaks down with entropy spikes and garbled text. Understanding and naming this mechanism is a prerequisite for anyone trying to run RL training in reduced precision reliably.

Who it affects

Teams and researchers building or scaling reinforcement learning pipelines for large language models, particularly those trying to use FP8 precision to cut training cost or time. The experiments span model scales from 8B to 32B parameters and two RL algorithms, GRPO and DAPO, suggesting the problem and the fix generalize across a meaningful range of setups rather than being specific to one model size or algorithm.

How to use it

The paper proposes Calibrated Clipping as a drop-in dynamic correction: it adjusts FP8 clipping bounds to track high-precision BF16 distributions during training, matching the lower-bound clipping quantile and rebalancing the upper bound. No pricing, license, or release details are given; this is a research method described in the paper rather than a shipped tool.

How solid is it

The claims come from the paper's own abstract and description of its experiments, which the authors describe as extensive: multiple algorithms (GRPO, DAPO), a range of model scales (8B to 32B), and multiple FP8 scaling granularities. The source text does not include specific numeric results, datasets, or benchmarks for the FP8-versus-BF16 comparison, nor does it name the authors, their institution, or a publication date.

Risks and caveats

Because the available text omits concrete performance numbers, datasets, and authorship, the strength of the improvement and its reproducibility cannot be independently assessed from what is given here. The described failure mode, gradients being erroneously zeroed for negative-advantage tokens, implies that any FP8 RL setup without a comparable correction may be silently under-penalizing bad model behavior until it surfaces as visible instability.

“compounded FP8 quantization noise distorts the importance ratio, disproportionately pushing negative-advantage tokens outside the trust region and erroneously zeroing out their gradients”

— the authors