Glonce

PACT sharpens critic training for reinforcement learning in LLMs

researchThe Glonce editorsHugging Face🔥72Glonce rating72Impact68Novelty75Relevance100Surprise40Players25Credibility85
PACT sharpens critic training for reinforcement learning in LLMs

A research paper addresses a gap in reinforcement learning for large language model post-training: there has been no generally accepted mathematical definition of token-level credit, the signal that tells a model which of its individual tokens deserve reward or blame. The authors formulate three regularity conditions, Completeness, Prefix Consistency, and Neutrality, and prove these three conditions together uniquely determine token-level credit. This gives a single mathematical basis for explaining behavior across existing RL training algorithms rather than treating each one as a separate heuristic.

Applying this framework, the authors show that an ideal teacher in On-Policy Distillation acts as an implicit critic, producing an expected policy gradient proportional to the one induced by proper token-level credit. They also find that response-level REINFORCE Leave-One-Out (RLOO) signals, despite being coarser than token-level signals, still match the expected policy-gradient contribution of token-level credit. Separately, they establish that under bounded outcome rewards, credit is approximately sparse, and they show that in Generalized Advantage Estimation (GAE), intermediate critic errors can grow large enough to become comparable to the underlying credit signal itself, undermining training.

These findings motivate a new training procedure, Policy Aligned Critic Training (PACT), which updates the actor before the critic (an Actor-then-Critic order) and applies an importance sampling correction to critic training so the critic stays aligned with the just-updated policy. On agentic mathematical reasoning, evaluated across four benchmarks, PACT reaches 72.87% average accuracy, beating GRPO by 8.80 percentage points and PPO by 13.16 percentage points. On SWE-bench Verified, a coding benchmark, PACT reaches a 67.4% pass rate, ahead of PPO by 2.4 points, GRPO by 2.0 points, and a method called SAO by 3.8 points.

Key facts

  • The paper proves three regularity conditions, Completeness, Prefix Consistency, and Neutrality, uniquely determine token-level credit in RL for LLM post-training
  • PACT (Policy Aligned Critic Training) uses an Actor-then-Critic update order with an importance sampling correction to keep the critic aligned with the updated policy
  • On four agentic mathematical reasoning benchmarks, PACT reaches 72.87% average accuracy, beating GRPO by 8.80 percentage points and PPO by 13.16 percentage points
  • On SWE-bench Verified, PACT reaches a 67.4% pass rate, ahead of PPO by 2.4 points, GRPO by 2.0 points, and SAO by 3.8 points
  • The authors show intermediate critic errors in Generalized Advantage Estimation (GAE) can grow comparable to the underlying credit signal, motivating PACT's design

Why it matters

Token-level credit assignment, deciding which tokens in a generated response deserve reward or blame, has been an unformalized heuristic in RL post-training of LLMs. By proving a unique mathematical characterization and deriving PACT from it, the paper offers both a theoretical explanation for why some existing signals (like RLOO) work despite coarser granularity, and a concrete training method that outperforms established baselines like PPO and GRPO on reasoning and coding tasks.

Who it affects

This is aimed at researchers and engineers building RL post-training pipelines for LLMs, particularly for agentic tasks such as multi-step mathematical reasoning and software engineering agents evaluated on benchmarks like SWE-bench Verified.

How to use it

PACT is a training procedure, not a released model or product. Teams running actor-critic RL post-training could adopt its Actor-then-Critic update order and importance sampling correction for critic training in place of methods like PPO, GRPO, or GAE-based approaches, based on the reported gains on the paper's benchmark suite.

How solid is it

The source is the paper's own abstract-level description, which gives the theoretical claims and specific benchmark numbers but does not name the authors, their institutions, a publication date, the identities of the four math-reasoning benchmarks, or what the compared method SAO stands for.

Risks and caveats

The benchmark results are self-reported by the paper's authors and have not been independently verified here. Without author and institutional attribution or a publication date in the source, provenance and peer-review status cannot be assessed from this text alone.