Glonce

ERPO curbs LLM training drift by regularizing prompts, not answers

researchThe Glonce editorsHugging Face🔥73Glonce rating73Impact62Novelty77Relevance100Surprise52Players48Credibility85
ERPO curbs LLM training drift by regularizing prompts, not answers

Policy optimization for large language models faces what the authors call a stability versus exploration dilemma, usually handled with an action-side regularizer known as Policy-KL. Keeping Policy-KL constrains how far the model's responses can vary and eats into the exploration budget available on the output side. Dropping it removes any explicit control on how far training drifts from the starting policy. Either way, practitioners are stuck absorbing one of the two costs.

The authors propose moving the regularization to the input side instead. As training runs, the distribution of queries the current policy is trained on drifts away, unchecked, from its pre-reinforcement-learning reference distribution. Environment-Regularized Policy Optimization (ERPO) introduces a Query-KL (QKL) term that bounds how far this query distribution can move, paired with a per-query weight, fixed for the dataset and derived from the reference distribution, that biases each update toward queries typical under the reference. Because the QKL gradient flows only through the likelihood of the query itself, and the response score function used by policy-gradient estimators never enters the QKL term, the term applies no direct pressure to the response distribution, so exploration on the output side is preserved.

ERPO is built to drop into existing GRPO, PPO, and REINFORCE-style training pipelines in place of the Policy-KL regularizer, without requiring additional forward passes through the model. On six mathematical reasoning benchmarks, the authors report that ERPO controls query-distribution drift effectively while delivering stronger accuracy and substantially more stable behavior than the standard Policy-KL approach, including under high-temperature decoding and over long-horizon training runs. The article does not name the specific benchmarks or give numerical results. The authors have published source code at github.com/alibaba/ERPO.

Key facts

  • ERPO replaces the standard Policy-KL regularizer with an input-side Query-KL (QKL) term that bounds drift in the distribution of training queries rather than in model responses.
  • QKL gradients flow only through query likelihood, not through the response score function used by policy-gradient estimators, so the term applies no direct pressure to the model's output distribution.
  • ERPO adds a dataset-static, reference-derived per-query weight that biases updates toward queries typical of the pre-reinforcement-learning reference distribution.
  • It plugs into GRPO, PPO, and REINFORCE-style pipelines without additional forward passes.
  • On six mathematical reasoning benchmarks, the authors report stronger accuracy and more stable behavior under high-temperature decoding and long-horizon training; source code is at github.com/alibaba/ERPO.

Why it matters

Reinforcement learning for large language models has relied on regularizing the model's own responses to keep training stable, which the authors argue forces a trade-off: constrain the output enough to prevent drift, and exploration room shrinks with it, or drop the constraint and lose control over how far training moves from where it started. ERPO's proposal, to regularize the training queries instead of the responses, offers a way to bound drift without taxing the exploration budget on the output side, at least by the authors' own account of the problem.

Who it affects

The work targets researchers and engineers who build reinforcement-learning pipelines for large language models, in particular teams running GRPO, PPO, or REINFORCE-style policy optimization, where ERPO is designed to substitute directly for the existing Policy-KL regularizer.

How to use it

ERPO is described as a drop-in replacement: it fits into existing GRPO, PPO, and REINFORCE-style pipelines without adding forward passes through the model. The authors have published source code at github.com/alibaba/ERPO. No licence terms, pricing, or release version are stated in the source.

How solid is it

The claims rest on results across six mathematical reasoning benchmarks, where the authors report better accuracy and more stable behavior under high-temperature decoding and long-horizon training compared with the standard Policy-KL regularizer. The article text does not name the specific benchmarks or give numerical results, so the size of the improvement cannot be judged from the source alone.

Risks and caveats

The source text gives no author names, institutions, or publication venue, and no independent evaluation of ERPO beyond the authors' own reported benchmarks. The exploration-preserving property and the stability gains both come from the authors' own description of their method, and the reported results cover only mathematical reasoning tasks.

“ERPO plugs into GRPO/PPO/REINFORCE-style pipelines without additional forward passes”

— the paper