CoRT redistributes GRPO reward token by token instead of spreading it evenly

Rubric-based reinforcement learning grades a language model's output against a checklist of explicit criteria, but in GRPO-style training pipelines that structured judgment collapses into one scalar reward for the whole response. That single number becomes one advantage value, and it gets broadcast uniformly to every token the model generated, so there is no way to credit the specific span, formatting choice, or wording that actually satisfied a given criterion. Bo-Wen Zhang and co-authors propose CoRT (Counterfactual Replay for Token-level rubric-guided policy optimization) to fix this without adding a trained scoring model. CoRT takes the same sampled response and rescores it twice: once under the original rubric-conditioned prompt, and once under a matched prompt with the criteria stripped out. Comparing the token-by-token log-likelihoods between these two rescorings produces a per-token signal for how much each token actually depended on the rubric. CoRT converts these contrasts into bounded, response-normalized weights and uses them to redistribute the original signed GRPO advantage across the response's tokens, leaving the response-level reward itself untouched and requiring no auxiliary scorer or separate training stage. Tested across instruction-tuned models and different reward granularities, CoRT beat matched response-level GRPO in the vast majority of comparisons, with an average gain of 4.4 percentage points, and stayed competitive with baselines that use a separately learned token-level credit model, despite skipping that extra training stage entirely.

Key facts

Problem: GRPO-style rubric RL reduces a multi-criteria judgment to one scalar reward per response, broadcast evenly to all tokens with no way to credit specific spans
CoRT rescores the same sampled response under a rubric-conditioned prompt and a matched criteria-free prompt, then compares token-level log-likelihoods between the two
Those contrasts become bounded, response-normalized weights that redistribute the existing signed GRPO advantage across tokens, without an auxiliary scorer or a separate training stage
Across instruction-tuned models and reward granularities, CoRT beats matched response-level GRPO in the vast majority of comparisons, averaging a 4.4 percentage-point gain
CoRT stays competitive with baselines that use a separately learned token-level credit model, while avoiding that extra relevance-learning stage

Why it matters

Rubric-based reward is supposed to be precise: different criteria can point at different parts of a response, different formatting decisions, or different semantic choices. GRPO throws that precision away by collapsing everything into a single response-level number and spreading it evenly across every token, so a model has no direct signal for which of its choices earned or lost the reward. CoRT is an attempt to recover that lost granularity cheaply, using only the model's own likelihoods rather than a new scoring network.

Who it affects

Anyone training or fine-tuning language models with rubric-based reinforcement learning on top of GRPO-style pipelines, including instruction-tuning teams that grade outputs against structured criteria rather than a single preference score. It is a training-time technique for post-training researchers and engineers, not a consumer-facing product or tool.

How to use it

CoRT plugs into an existing rubric-conditioned GRPO setup: for each sampled response, it additionally rescores that same response under a criteria-free version of the prompt, computes the token-level log-likelihood contrast between the two, normalizes and bounds these contrasts into weights, and uses them to redistribute the already-computed signed advantage across tokens. It changes nothing about how the response-level reward itself is computed, and adds no separate trained model or training stage, only an extra rescoring pass on the same response.

How solid is it

The authors report results across instruction-tuned models and multiple reward granularities, finding CoRT ahead of matched response-level GRPO in the vast majority of comparisons, with an average gain of 4.4 percentage points, and roughly on par with baselines that rely on a separately learned token-level credit model. The paper does not specify which instruction-tuned models or exactly how many comparisons were run in the text provided, and discussion of the paper on Hugging Face is limited to a handful of comments so far.

Risks and caveats

A 4.4 percentage-point average gain across the vast majority of comparisons is not a win in every case, and the source text does not break down where CoRT underperforms or by how much. Running counterfactual replay means rescoring each sampled response twice, once with and once without the rubric context, and the text does not quantify the added compute cost of that second pass. As an academic proposal, it has not been reported at production scale or validated outside the paper's own experiments.

“These results suggest that policy-internal counterfactual likelihood contrasts provide an effective training signal for within-response credit allocation while retaining the simplicity and stability of GRPO.”

— CoRT paper, Bo-Wen Zhang and co-authors