Glonce

Researchers find 'Value Flattening' flaw in PPO critics, propose fix

researchThe Glonce editorsHugging Face🔥55Glonce rating55Impact60Novelty72Relevance95Surprise55Players35Credibility78
Researchers find 'Value Flattening' flaw in PPO critics, propose fix

Proximal Policy Optimization (PPO) is a standard reinforcement-learning algorithm for training large language models, and it relies on a critic network to estimate the value of intermediate states so that policy updates have less variance. The authors report a systematic failure in that critic, which they name Value Flattening: when they estimate the true state values by running multiple Monte Carlo continuations from each state, those values change sharply between neighboring intermediate states, but the critic's own predictions stay comparatively flat across the same states, failing to track the real swings. They reproduce the same pattern in a controlled FrozenLake environment, where it gets worse as the state space grows larger. Their theoretical and empirical analysis traces the cause to two factors: an implicit variance penalty baked into the standard critic loss, and redundant gradient updates from temporally correlated states that look similar to the critic and push it toward the same flat prediction. Based on this diagnosis, they propose SParse Proximal Policy Optimization (SP^3O), which applies the value loss to only a small number of well-separated states within each response instead of every state. In experiments on Qwen3-Base, SP^3O supervising just three states per response mitigates Value Flattening and consistently improves the resulting policy across different model sizes and evaluation suites. The paper does not report a specific numeric performance gain over standard PPO, name the model sizes or evaluation suites tested beyond Qwen3-Base, or identify individual authors or institutions.

Key facts

  • PPO critics used in LLM reinforcement learning show 'Value Flattening': true state values from Monte Carlo continuations swing sharply between states, but the critic's predictions stay comparatively flat.
  • The same pattern shows up in a controlled FrozenLake environment and worsens as the state space grows.
  • The cause is traced to an implicit variance penalty in the critic loss plus redundant updates from temporally correlated, similar-looking states.
  • The proposed fix, SP^3O, applies the value loss to only a few well-separated states per response instead of every state.
  • On Qwen3-Base, supervising just three states per response mitigates Value Flattening and improves the learned policy across model sizes and evaluation suites.

Why it matters

PPO's critic is supposed to make policy training more stable by giving an accurate read on how good an intermediate state is. If the critic instead flattens out real swings in value, as this paper argues happens systematically, then the variance reduction it is meant to provide is partly an illusion, and training signal quality suffers in ways that are easy to miss because the critic still produces plausible-looking numbers.

Who it affects

The finding concerns anyone training large language models with PPO-style reinforcement learning, since the critic component described is a standard part of that pipeline, not a special case. The experiments are run on Qwen3-Base and a FrozenLake toy environment, so the direct evidence covers those setups specifically.

How to use it

The proposed method, SP^3O, is a drop-in change to how the critic loss is computed: instead of applying it to every state in a response, it is applied to only a few well-separated ones, three in the reported Qwen3-Base experiments. This is described as mitigating both root causes the authors identify, the implicit variance penalty and the redundant correlated-state updates, without needing a different critic architecture.

How solid is it

The paper backs its claim with two lines of evidence: a theoretical analysis connecting Value Flattening to the critic loss's implicit variance penalty and to correlated-state gradient redundancy, and empirical results in both a controlled FrozenLake environment and Qwen3-Base training. It reports that SP^3O improves the learned policy consistently across model sizes and evaluation suites, though it does not give a specific numeric margin over standard PPO in the material available here.

Risks and caveats

The source text does not specify which model sizes or evaluation suites were tested beyond Qwen3-Base, nor does it give a numeric performance gain, so the strength of the improvement cannot be quantified from what is stated. It also does not name individual authors or institutions or state a publication venue or date.