Glonce

AI benchmarks are contaminated and gamed, new studies show

researchThe Glonce editorsHacker News🔥61Glonce rating61Impact75Novelty65Relevance100Surprise65Players78Credibility82

BIG-bench, a benchmark suite built by hundreds of researchers, ships with a unique "canary" string: a tracking marker meant to let anyone training a model filter the benchmark out of their corpus, and let anyone auditing a model check whether they failed to. When OpenAI ran its contamination checks for the GPT-4 technical report, it found that BIG-bench had been swallowed into GPT-4's training data anyway, forcing it to exclude the results; the model could reproduce the canary on request. The post uses that episode to reintroduce Goodhart's Law: Charles Goodhart wrote in 1975, about British monetary policy, that once a measure becomes a target, it stops being a good measure. Fifty years later, the post argues, benchmark scores decide which AI companies raise money, which models enterprises buy, and which press releases get written, so the incentive to hit the number has never been stronger.

The post frames this as a pattern computing keeps rediscovering: Campbell's Law and No Child Left Behind showed that once standardized test scores became the target, teachers taught to the test, and scores rose while learning did not; machine translation spent years optimizing the BLEU metric into systems that scored well but translated badly, because BLEU rewards n-gram overlap rather than meaning. Manheim and Garrabrant formalized four variants of "Goodharting" in 2018; the post says two now dominate AI benchmarking and each has quantitative evidence behind it: regressional Goodharting, where the test quietly becomes part of the training data, and adversarial Goodharting, where the evaluation process itself gets gamed.

The contamination mechanism itself is mundane, per the post: benchmarks live on the public web (GitHub repositories, papers, Stack Overflow threads, derivative datasets), web-scale pretraining scrapes all of it, and a "held-out" test set that has been public for three years is not actually held out; nobody has to decide to cheat, the post argues, because the pipeline cheats by default. The cleanest demonstration cited is GSM1k, from Scale AI (Zhang et al., 2024): researchers commissioned 1,205 fresh grade-school math problems, written by human annotators and matched to the difficulty distribution of GSM8K (the standard benchmark for elementary math reasoning), then re-tested leading models on questions no crawler had ever seen. Accuracy fell by as much as 13% for the worst offenders in the paper's initial evaluation, and by up to 8% in the final version run against the full released set; the Phi and Mistral model families showed what the paper's authors called systematic overfitting across almost all sizes. The giveaway was a correlation: the more likely a model was to reproduce GSM8K problems verbatim, the larger its score gap between GSM8K and GSM1k, a Spearman's r² of 0.36.

A second strand of evidence says the benchmarks themselves are error-riddled. Gema et al. re-annotated 5,700 MMLU questions across all 57 subjects and estimate that 6.49% of the benchmark contains errors: wrong answer keys, ambiguous phrasing, unanswerable questions. In the virology subset specifically, 57% of the questions they analyzed were flawed, and correcting the errors changed model rankings. Truong et al.'s "Fantastic Bugs" audit applied measurement-theoretic statistics across nine widely used benchmarks and found, on expert review, that up to 84% of the top flagged questions had substantive flaws. GSM8K's own baseline error rate runs around 5%, per the post.

The third strand covers Chatbot Arena, the live leaderboard where anonymized models are pitted against each other and real-user votes feed a Bradley-Terry ranking model, built specifically to avoid a fixed, leakable question set. "The Leaderboard Illusion" (Singh et al., 2025), from researchers at Cohere Labs, Stanford, Princeton, MIT, Ai2 and elsewhere, analyzed roughly two million Arena battles across 243 models and 42 providers. It reports that Meta tested 27 private Llama 4 variants on the Arena before launch, then published only what it chose to; selecting the best of many attempts, the authors show, breaks the statistical assumptions the rankings depend on. To measure the resulting noise, the researchers submitted two identical checkpoints of the same model under different names, and their scores diverged by 17 points. The paper also finds a data asymmetry: the top two proprietary providers each received an estimated 19% to 20% of all Arena battle data individually, while 83 open-weight models combined received under 30%. In controlled experiments, fine-tuning on Arena-distribution data produced relative gains of up to 112% on ArenaHard, a same-distribution test set, while MMLU performance slightly declined; as the post puts it, models were learning to win Arena, not to be better. Separately, and without citing a specific source, the post states that Meta was caught submitting a Llama 4 variant optimized for "conversationality" that outranked the model it actually shipped.

LMArena disputed the framing, and the post credits parts of the rebuttal: the identical-checkpoint scores fall within overlapping confidence intervals, the private-testing policy had been public since March 2024 and open to any provider with the capacity to use it, and the 112% figure comes from ArenaHard rather than the live Arena. The post argues, though, that none of this disputes the core finding: a handful of well-resourced labs systematically used best-of-many submission while smaller labs submitted once, and the ranking model assumes nobody does that.

The post traces an incentive chain: leaderboard position drives press coverage, press coverage feeds fundraising decks, and decks shape enterprise procurement. Nobody in that chain is paid to ask what the number actually means, a dynamic the post calls "benchmarketing." None of this requires fraud: providers can legitimately choose the temperature, prompting strategy and few-shot configuration that maximize a headline number, settings that rarely match production defaults; each choice is defensible on its own, but the post says the sum is a score that describes a model nobody will ever actually run. Saturation compounds it: MMLU has gone from genuinely hard to a benchmark frontier models pass at roughly 90%, and HumanEval and HellaSwag are effectively ceilinged, so the field keeps moving to new benchmarks that then contaminate and saturate in turn. Teams that build evaluations from their own legal, medical or financial workloads, the post says, routinely find real accuracy well below what the nearest public benchmark implied.

The post's constructive section is deliberately modest about fixes. Private, periodically refreshed test sets attack contamination directly; GSM1k is offered as the template, echoing how vision researchers a decade earlier used ImageNetV2 and CIFAR-10.1 to expose that image classifiers had overfit the original datasets, but such sets are expensive to build and still "rot," just more slowly. Contamination-detection methods, like n-gram overlap checks and perplexity probes, should be standard practice but are not a fix: Sun et al. tested 20 proposed mitigation strategies across 10 models and five benchmarks and found none significantly improved contamination resistance over doing nothing, and none stayed faithful to the original benchmark while resisting contamination. The most practical advice offered is to evaluate models on your own data, a 20-to-50 question mini-benchmark built from real tickets, contracts or queries, since no public model has trained on that specific workload, while guarding against the biases of any LLM judge used to scale grading. For shared leaderboards, the post points to structural fixes proposed by the Leaderboard Illusion authors: prohibit score retraction, require disclosure of every variant tested, cap private submissions, and equalize sampling, while expressing doubt that any leaderboard can hold that line while its biggest users are also its biggest names.

Key facts

  • OpenAI's own contamination checks for the GPT-4 technical report found that BIG-bench, a benchmark suite carrying a tracking "canary" string meant to keep it out of training data, had been swallowed into GPT-4's training set anyway, and its results had to be excluded.
  • GSM1k (Scale AI, Zhang et al., 2024) retested leading models on 1,205 fresh grade-school math problems matched to GSM8K's difficulty distribution; accuracy fell by as much as 13% for the worst offenders in the initial evaluation and by up to 8% in the final version, with the Phi and Mistral families showing the worst overfitting.
  • Gema et al. re-annotated 5,700 MMLU questions across all 57 subjects and estimate that 6.49% of the benchmark contains errors, rising to 57% among the virology questions they analyzed; correcting the errors changed model rankings.
  • "The Leaderboard Illusion" (Singh et al., 2025) found Meta tested 27 private Llama 4 variants on Chatbot Arena before launch and published only what it chose to, and that two identical model checkpoints submitted under different names scored 17 points apart.
  • Sun et al. tested 20 proposed contamination-mitigation strategies across 10 models and five benchmarks and found none significantly improved resistance over doing nothing.

Why it matters

Public benchmark scores are not an academic curiosity: per the post, they decide which AI companies raise money, which models enterprises buy, and which press releases get written, so a compromised scoring system misdirects real capital and adoption, not just leaderboard bragging rights. What makes this more than a familiar complaint is that the failure is not singular: contamination (test data leaking into training, as with BIG-bench and GSM8K) and gaming (leaderboard manipulation, as documented on Chatbot Arena) are separate mechanisms that, per the post, now both have quantitative studies behind them, unlike earlier, anecdote-level critiques such as BLEU in machine translation or teaching to standardized tests.

Who it affects

AI labs whose leaderboard position feeds fundraising decks and press coverage; enterprise procurement teams choosing a model on the strength of a public score; benchmark maintainers and researchers whose test suites get contaminated or gamed within a release cycle or two; and end users and business customers who, per the post, often find real-world accuracy on their own legal, medical or financial workloads well below what the nearest public benchmark implied. OpenAI and Meta both appear as direct examples: OpenAI's own contamination checks caught BIG-bench inside GPT-4's training data, and Meta is reported, per "The Leaderboard Illusion," to have tested 27 private Llama 4 variants on Chatbot Arena before publishing only what it chose to.

How to use it

The post's most concrete recommendation is to evaluate models on your own data: a 20-to-50 question mini-benchmark built from actual tickets, contracts or queries, since no public model has trained on that specific workload. If an LLM judge is used to scale the grading, it warns to control for the judge's known biases toward position, verbosity and its own outputs, or the private eval gets Goodharted too. Beyond that, it recommends favoring private, periodically rotated test sets like GSM1k over static public ones, treating contamination-detection checks such as n-gram overlap and perplexity probes as routine hygiene rather than a fix, and pushing shared leaderboards toward the structural changes proposed by the authors of "The Leaderboard Illusion": prohibit score retraction, require disclosure of every variant tested, cap private submissions and equalize sampling across providers.

How solid is it

The post is a synthesis of several distinct, named quantitative studies rather than an argument resting on its own authority: Zhang et al.'s GSM1k paper (Scale AI, 2024), Gema et al.'s MMLU re-annotation, Truong et al.'s "Fantastic Bugs" audit, Sun et al.'s test of 20 mitigation strategies, and Singh et al.'s "The Leaderboard Illusion" (2025), a multi-institution paper from Cohere Labs, Stanford, Princeton, MIT, Ai2 and elsewhere. On the Arena-gaming claims, the post also presents LMArena's rebuttal and credits parts of it as landing: the identical-checkpoint score gap falls within overlapping confidence intervals, and the 112% figure is an ArenaHard number rather than a live-Arena one. It argues, though, that the rebuttal leaves the core asymmetry unaddressed: well-resourced labs submitted best-of-many while smaller labs submitted once. One further claim, that Meta separately submitted a Llama 4 variant optimized for "conversationality," is asserted without a named source. The piece itself runs on cacm.acm.org's blog rather than the peer-reviewed Communications of the ACM (CACM) magazine, and carries no publication date of its own; its end-of-article bio names Alex Williams, described there as a full-stack developer and former Hosting Data U.K. owner turned independent IT consultant and technical copywriter, rather than a credentialed ML researcher, so it reads as a synthesis and argument built on other groups' published work.

Risks and caveats

The post does not name which specific models, beyond "the Phi and Mistral families," were re-tested in the GSM1k evaluation, or who caught Meta's "conversationality"-optimized Llama 4 variant, or when. It gives the 112% figure only as a relative gain, without the absolute ArenaHard or MMLU scores behind it, and does not name the "top two proprietary providers" in the Arena data-asymmetry finding. It also does not quantify how long a rotated private test set stays uncontaminated, only that rotation makes it "rot slower." Because this is a synthesis rather than original research, and a few of its most pointed claims are asserted without a named source, readers who need exact figures should check the underlying papers directly: GSM1k, the MMLU re-annotation, "Fantastic Bugs" and "The Leaderboard Illusion."

“Treat every public score as a marketing claim that happens to carry decimal places.”

— Alex Williams, in the CACM blog post