Glonce

BPCO trains a stable critic that matches GRPO on one sample

researchThe Glonce editorsHugging Face🔥71Glonce rating71Impact60Novelty70Relevance100Surprise55Players25Credibility87
BPCO trains a stable critic that matches GRPO on one sample

Most current reinforcement learning recipes for large language models, such as GRPO, sidestep the need for a trained critic by sampling several responses to the same prompt and comparing them against each other to estimate advantages. Penghui Qi and co-authors argue this is a workaround for a real problem: a reliable critic could estimate token-level advantages from a single response instead, but standard recipes for training such a critic tend to be unstable in practice, which is why group-based sampling became the default. The paper studies that instability directly and proposes Best Practice Critic Optimization (BPCO), a recipe that combines five specific design choices: DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation. Because the critic is used only during training and never at inference, BPCO can also feed it reward-defining information that stays hidden from the policy itself, such as a reference answer or a grading rubric. The authors ran controlled experiments that isolate the effect of each of the five design choices individually. Testing spanned mathematical reasoning tasks across models ranging from 1.5B parameters up to a 30B-A3B mixture-of-experts model. Across that range, BPCO consistently improved on a strong critic-based baseline, and matched or exceeded a group-based baseline despite sampling only one response per prompt instead of several. The same recipe also improved learning when rewards were rubric-based rather than tied to a single reference answer. The authors conclude that a carefully engineered critic is a reliable alternative to group-relative advantage estimation. Code implementing BPCO is publicly available at github.com/QPHutu/golden_critic.

Key facts

  • BPCO combines five techniques: DPPO, value predictions bounded to the reward range, Monte Carlo value targets, unnormalized policy advantages, and length-adaptive generalized advantage estimation.
  • Tested on mathematical reasoning tasks across models from 1.5B parameters up to a 30B-A3B mixture-of-experts model.
  • BPCO matches or exceeds a group-based baseline (like GRPO) while sampling only one response per prompt instead of several.
  • Because the critic runs only during training, it can be conditioned on hidden reward information such as a reference answer or grading rubric.
  • The same recipe also improves learning when rewards come from a grading rubric rather than a single reference answer, and the code is public on GitHub.

Why it matters

Group-based methods like GRPO became popular partly because training a stable critic was hard, so they trade that difficulty for sampling multiple responses per prompt, which costs more compute per training step. BPCO attacks the original problem directly: it engineers a critic stable enough to estimate advantages from a single response, closing the case for why group sampling was preferred in the first place.

Who it affects

Researchers and engineers building reinforcement learning pipelines for large language models, especially teams working on mathematical reasoning or other tasks graded against a reference answer or rubric, where BPCO's ability to hide reward-defining information from the policy while still using it in the critic is directly applicable.

How to use it

The authors have published code implementing BPCO at github.com/QPHutu/golden_critic. No pricing, license terms, or product packaging are mentioned; this is a research recipe rather than a hosted tool or service.

How solid is it

The claims rest on controlled experiments that isolate the contribution of each of the five design choices individually, across a range of model sizes from 1.5B to a 30B-A3B mixture of experts, on mathematical reasoning tasks. The paper's own text does not name the specific benchmarks or datasets used, nor does it state numeric accuracy or score improvements; those figures were not available in the source material reviewed for this piece.

Risks and caveats

The source text gives no institution or affiliation for the authors and no publication or submission date. Without named benchmarks or reported numbers, the strength of the improvement over group-based and critic-based baselines cannot be independently sized from the text alone; readers who need exact figures should consult the paper directly.

“These results show that a carefully designed critic provides a reliable alternative to group-relative advantage estimation.”

— the paper's authors