Glonce

RubricForge roughly halves false-pass rate in agent judging

researchThe Glonce editorsarXiv🔥73Glonce rating73Impact70Novelty75Relevance95Surprise55Players20Credibility85

Grading language-model agent trajectories at scale increasingly relies on a second LM as an automatic judge, because the gold signal, an executable environment reward, is expensive, slow, or unavailable at deployment time. The authors note that existing judges either hand-write the scoring rubric, as G-Eval does, or fine-tune the judge's weights, and both tend to credit fluent but unsuccessful trajectories as successes. RubricForge takes a third route: it induces the text of a judging rubric from a small set of ground-truth-labeled trajectories through reflective evolution, optimizing the rubric to agree with the environment reward, then freezes it and applies it to new, held-out trajectories in a single model call with no environment access required at judging time. Because the optimized rubric is human-readable text rather than model weights, every verdict traces back to named criteria. The authors tested the method using one frozen 7B model as both the agent and the judge, on tau-bench (173 labeled trajectories drawn from 220 rollouts) and WebShop (160 labeled trajectories), comparing it against a generic G-Eval judge. On raw agreement with the ground-truth reward, RubricForge's edge over G-Eval was not statistically significant (McNemar p = 0.248), and on absolute-score calibration error the generic G-Eval judge was marginally favored (|err| difference -0.048, p = 2x10^-4). The paper's central claim is that the real gain sits elsewhere: on tau-bench, RubricForge over-credited failed trajectories roughly half as often as G-Eval, with a false-pass rate of 0.115 versus G-Eval's 0.173, including three cases where RubricForge caught over-crediting that G-Eval missed and zero cases where it introduced new over-crediting of its own. On WebShop, RubricForge also ranked graded outcomes more faithfully, with a Spearman correlation of 0.410 against G-Eval's 0.370. The authors argue that for a reward-free evaluator, the false-pass rate rather than aggregate agreement is the metric that matters for deployment, since a false pass ships a broken agent while a false fail only costs a retry.

Key facts

  • RubricForge induces a human-readable judging rubric from a small set of ground-truth-labeled trajectories via reflective evolution, then freezes it and applies it to new trajectories in one model call with no environment access.
  • Tested with a single frozen 7B model as both agent and judge, on tau-bench (173 labeled trajectories from 220 rollouts) and WebShop (160 labeled trajectories).
  • RubricForge's edge over a generic G-Eval judge on raw agreement is not statistically significant (McNemar p = 0.248), and G-Eval's absolute-score calibration is marginally better (|err| difference -0.048, p = 2x10^-4).
  • On tau-bench, RubricForge's false-pass rate is 0.115 versus G-Eval's 0.173, roughly half as often, with three over-credit catches and zero reversals.
  • On WebShop, RubricForge ranks graded outcomes more faithfully than G-Eval: Spearman correlation 0.410 versus 0.370.

Why it matters

LM-based judges stand in for expensive or unavailable environment rewards when evaluating agent trajectories at scale, but the authors say existing approaches, whether a hand-written rubric like G-Eval or a fine-tuned judge, tend to credit fluent but unsuccessful trajectories as successes. RubricForge targets that specific failure by grounding the rubric's text in real outcome labels through reflective evolution rather than hand-authoring or opaque weight tuning, and keeps the result as readable text so each verdict traces to a named criterion.

Who it affects

Teams building or evaluating language-model agents, such as tool-use or web-shopping agents, that rely on an LM judge instead of a live environment reward during development or at deployment, and researchers designing agent benchmarks who need a more faithful proxy for task success.

How to use it

RubricForge is evolved against a modest set of ground-truth-labeled trajectories, 173 for tau-bench and 160 for WebShop, then frozen and applied to new, held-out trajectories in a single model call with no environment access needed at judging time, so the same 7B model used as the agent can also serve as the judge.

How solid is it

Results against the generic G-Eval baseline are mixed. Overall agreement with the ground-truth reward is not statistically significant (McNemar p = 0.248), and G-Eval's absolute-score calibration is marginally better (|err| difference -0.048, p = 2x10^-4). The paper's own framing is that the gain is in faithfulness rather than raw agreement: RubricForge's false-pass rate on tau-bench is 0.115 against G-Eval's 0.173, with three cases where it caught over-crediting that G-Eval missed and none where it added new over-crediting, and it ranks WebShop outcomes more faithfully (Spearman 0.410 versus 0.370).

Risks and caveats

The false-pass improvement is not paired with a stated significance test in the source, and the one comparison that was tested for significance, raw agreement, came back non-significant, so the headline gain rests on a single axis the authors chose to foreground. Both benchmarks used one frozen 7B model as agent and judge, and the labeled sets used to induce the rubric are small, 173 and 160 trajectories. The source text names no authors or institutions, gives no publication date, and does not explain the mechanics of reflective evolution beyond naming it.