Glonce

GPT-OSS and cheap LLMs match Claude, Gemini as proof judges

researchThe Glonce editorsarXiv🔥69Glonce rating69Impact65Novelty55Relevance98Surprise48Players55Credibility78

Grading natural-language math proofs, checking a candidate proof against a ground-truth proof and a human grading rubric, is a routine and expensive step in evaluating math-reasoning AI systems, and the usual approach relies on costly frontier LLM judges. A new study asks whether cheap open-weight models can do the job just as reliably.

On a 200-instance validation sample from IMO-GradingBench, a benchmark for grading proofs against International Mathematical Olympiad-style problems, the authors tested three cheap open-weight judges: GPT-OSS 120B, DeepSeek-V4 Flash and Gemma-4 31B. All three agreed with human pass or fail decisions at rates statistically indistinguishable from two frontier judges, Claude Opus 4.7 and Gemini 3.1 Pro, while costing up to 100 times less.

The authors had expected that combining the three cheap judges by majority vote would produce the best low-cost option. It did not: the majority vote matched the frontier's accuracy but failed to improve on the single strongest of the three judges on its own.

Extending the comparison to the full 1000-instance IMO-GradingBench benchmark, the authors tested several ways of combining the three cheap judges' verdicts. Requiring all three judges to agree that a proof passes, an all-three-pass rule, produced the highest pass-agreement and precision of any rule tested, and, across four replicate runs, the smallest run-to-run variation.

The paper's headline finding is that cheap LLM judges are competitive with frontier models at one to two orders of magnitude lower cost. The authors recommend all-three-pass as a deployable default, but caution that the rule was identified after the fact, by testing options on the same data used to evaluate them, and needs independent replication before it can be trusted as a general default.

Key facts

  • On a 200-instance validation sample, three cheap open-weight judges (GPT-OSS 120B, DeepSeek-V4 Flash, Gemma-4 31B) matched frontier judges Claude Opus 4.7 and Gemini 3.1 Pro in agreement with human grading, at up to 100x lower cost.
  • A majority vote of the three cheap judges matched the frontier's accuracy but did not improve on the single best cheap judge, contrary to the authors' expectation.
  • On the full 1000-instance benchmark, requiring unanimous agreement among all three cheap judges (all-three-pass) gave the highest pass-agreement and precision, and the smallest run-to-run spread across four replicate runs.
  • The authors report cheap judges are competitive with frontier models at one to two orders of magnitude lower cost overall.
  • The authors recommend all-three-pass as a deployable default but note it was chosen post-hoc and needs independent replication.

Why it matters

Frontier LLM judges are the standard tool for grading free-form math proofs at scale, but they are expensive, which limits how much evaluation researchers building math-reasoning systems can afford to run. This study finds that cheap open-weight judges, GPT-OSS 120B, DeepSeek-V4 Flash and Gemma-4 31B, can do the same grading job at up to 100 times lower cost on the initial test, and one to two orders of magnitude lower cost overall, without a statistically detectable drop in agreement with human judgments. If the result holds up, it changes the economics of evaluating math-reasoning models.

Who it affects

Researchers and labs building and evaluating math-reasoning LLMs, and anyone running IMO-GradingBench or similar proof-grading pipelines who currently pays for frontier-model judging with Claude Opus 4.7 or Gemini 3.1 Pro.

How to use it

Deploy the three cheap judges, GPT-OSS 120B, DeepSeek-V4 Flash and Gemma-4 31B, together with an all-three-pass consensus rule: count a proof as passing only when all three judges independently agree it passes. In the authors' tests this rule beat both majority voting and any single judge on pass-agreement, precision and run-to-run consistency.

How solid is it

The comparison spans a 200-instance validation sample and the full 1000-instance IMO-GradingBench benchmark, with the all-three-pass rule checked across four replicate runs for stability. The authors themselves flag a limit: the all-three-pass rule was chosen after seeing which option performed best on this data, not fixed in advance, so its advantage needs independent replication to confirm it generalizes.

Risks and caveats

The rule that looks best here was identified post-hoc, the classic risk of overfitting an evaluation choice to the same data used to test it, and the authors explicitly call for independent replication before treating it as a safe default. The abstract gives no absolute accuracy or agreement-rate percentages, no dollar costs, and no detail on what the human-grading rubric or the construction of IMO-GradingBench actually involve, which limits how far the claim of matching frontier judges can be checked independently.

“We had expected a majority vote of the three to be the best budget option; it matched the frontier but did not improve on its strongest member.”

— the paper's authors