Glonce

Hugging Face audits ICML 2026 papers with AI agents, breaks a spotlight proof

researchThe Glonce editorsHugging Face🔥82Glonce rating82Impact78Novelty82Relevance95Surprise68Players77Credibility87
Hugging Face audits ICML 2026 papers with AI agents, breaks a spotlight proof

ICML 2026 received 23,918 submissions and accepted 6,352 papers, roughly double the previous year's total, a jump Hugging Face and alphaXiv say is at least partly driven by AI agents making it faster to run experiments and write them up. Review capacity has not doubled along with it. Reviewers at most conferences are volunteers who may not have the time or expertise to check a paper fully, and the post opens with a real comment on an accepted ICML 2026 spotlight paper: "My low confidence score is because I did not check all the proofs carefully." The organizers say they will come back to that exact paper later. What has changed, they write, is that coding agents such as Claude Code, Codex, Cursor and Pi can now read a paper, write the code, launch the experiments and report back on what they found; checking a paper carefully used to cost a reviewer a weekend, and an agent can now attempt it in an afternoon, in parallel, thousands of times over.

Rather than audit ICML 2026 themselves, Hugging Face and alphaXiv opened the task to the whole community, embracing whatever mix of agent frameworks, compute budgets and scientific judgment participants brought. From July 15 to August 2, 2026, they ran the ICML 2026 Open Reproductions challenge: a streamlined interface let a participant's agent pull a paper, its claims and the challenge instructions with a single command, then try to verify or falsify each claim through real experiments. Each participant received $20 in Hugging Face compute credits to run experiments on HF Jobs, and across the challenge participants launched 2,962 cloud jobs in total. Where a full reproduction was impossible, for example when a paper's dataset was proprietary or its checkpoints unreleased, participants ran toy reproductions on synthetic data built to mimic the original data's properties. The challenge drew 1,221 participants in total, which the organizers call probably the largest attempted reproduction of a scientific conference.

Aggregating the claim-level verdicts by paper: 51% of examined papers (1,103) had at least one claim independently verified. Of those, 266 papers were fully reproduced, with every extracted claim verified, and 632 more were partially reproduced with nothing falsified; in total, 3,978 individual claims were confirmed through real experiments. Another 23% of examined papers (496) had at least one claim falsified or contested: 49 of those had every claim falsified with nothing verified, and 242 of them saw independent reproduction teams reach opposite verdicts on the same claims, which the organizers take as evidence that "reproducibility is not binary; it is adversarial." The rest split between 502 papers with only toy-scale evidence and 280 papers where nothing could be established either way, most often because the needed artifacts were missing. The post's own title claims 2,200 papers were reproduced, but these four figures add up to 2,381; the piece does not explain the gap. Thirty-five participants formally claimed to have falsified a paper's claim, and Hugging Face says it adversarially re-verified every one of them, by re-reading the paper, re-reading the reproduction logbook, and re-deriving the math or re-implementing the experiment from the paper's own text.

Several confirmed falsifications are described in detail. The paper from the opening, "Towards Optimal Robustness in Learning-Augmented Paging," claims a specific robustness bound for its algorithm. One participant's logbook measured the bound's additive error term growing faster than claimed and located the exact step in the proof where it breaks; Hugging Face's own re-implementation extended the test to k = 1,024 and confirmed the faster growth at roughly nine sigma, meaning the paper's claimed bound does not hold. In "Attention's forward pass and Frank-Wolfe," which proves that token particles collapse to the origin whenever the origin starts inside their convex hull, three independent teams found counterexamples, with violations first appearing at step 224, roughly step 3,800, and step 6,416. The staggered timing explains why other checks had "verified" the claim: checks that stopped early never ran long enough to see the violation. The clearest of the three counterexamples is stated in exact rational arithmetic, ruling out floating-point ambiguity, and the paper's authors confirmed the problem the same day and are working on a fix. In "Self-Distillation Enables Continual Learning," the paper's central equation and its whole theory section analyze reverse KL divergence, but the released code's default setting, which the authors say produced all of the paper's results, actually computes forward KL divergence instead. The same logbook that caught the mismatch also could not reproduce the paper's headline result, a 4 percentage point improvement, even running the authors' own code on the authors' own data; the authors have since uploaded a clarified version of the paper to arXiv. In "Do Transformers Need Three Projections?," a participant found that about 66% of the evaluated label positions were EOS (end-of-sequence) padding tokens that train to near-zero loss, which deflated the measured perplexity by roughly threefold. Corrected for that padding, the abstract's claimed "3.1% quality cost for 50% cache reduction" becomes roughly 9.4%.

Hugging Face also caught a false positive running the other way. One logbook claimed dramatically that "the paper's method is 2x slower than the baseline." That turned out to be an arithmetic bug in the reproduction itself, comparing per-trajectory time against per-batch-of-50 time. Normalized correctly, the same participant's own data confirmed the paper's original claimed 8x speedup.

Hugging Face says it has begun writing to the authors of every confirmed finding with a simple pitch: here is what we found, here is the evidence, do you agree, or is the analysis wrong? Early responses have been positive. Authors have confirmed findings on multiple papers, two arXiv corrections are in progress, and in one case an author had already quietly fixed the error in a newer arXiv version about a month before the challenge caught it, which Hugging Face counts as independent convergence rather than a miss. The organizers describe the challenge as probably the largest attempted reproduction of a scientific conference and the largest open, claim-by-claim audit of a machine learning conference to date, and say they hope it will not hold that record for long. Every logbook, verdict, trace and artifact from the challenge is public, starting from the challenge's own Space, and the organizers say further reproduction events are planned, without giving a date.

The post's central question is whether humans still have a role in reviewing papers once agents can run the checks. The organizers argue yes, for two reasons. First, pure agent execution hits real limits: agents got stuck in local loops, misread behavior that only shows up at larger scale (several of the paging paper's "verified" verdicts came from checks that stopped before the slower-than-claimed growth became visible), and in one case built an entire falsification on a plain units mismatch. The most reliable results in the challenge came from workflows where a human actively steered the agent: re-pointing it, questioning an assumption, or stopping a run before it burned a week of compute on a premise that was already wrong. Second, some evaluation stayed irreducibly human. The clearest example is the challenge's "human-in-the-loop winner": a paper claiming stable image generation under extreme quantization, where the numerical metrics said "no collapse" but whether the resulting images were actually usable was a perceptual call. The participant's agent built a purpose-built review interface, and the human personally judged all 128 image pairs, committing the annotations to the repository for the agent to validate afterward; the published trace even shows the participant asking how the review tool works, then reporting back, "I have gone over the pairs and put the csv in the repo, please check." The organizers conclude that the human reviewer's job now is to manage the agent well, the way a professor or principal investigator sets up compute, tooling, data access and feedback for a graduate student. The participants who got the most out of their agents were the ones who built the right environment and asked the right questions, then let the agent do the running.

Key facts

  • Hugging Face and alphaXiv ran the ICML 2026 Open Reproductions challenge from July 15 to August 2, 2026: 1,221 participants, using whatever mix of AI coding agents they brought, each received $20 in Hugging Face compute credits, and together launched 2,962 cloud jobs.
  • 51% of examined papers (1,103) had at least one claim independently verified, with 3,978 individual claims confirmed overall; 23% (496) had at least one claim falsified or contested, including 49 papers where every claim was falsified and 242 where independent teams reached opposite verdicts on the same claims.
  • The claimed robustness bound in the ICML 2026 spotlight paper "Towards Optimal Robustness in Learning-Augmented Paging" did not hold: Hugging Face's own re-implementation extended the test to k = 1,024 and confirmed faster-than-claimed growth at roughly nine sigma.
  • "Self-Distillation Enables Continual Learning" turned out to compute the wrong KL divergence in its released code and could not reproduce its own headline 4-percentage-point result, even on the authors' own code and data; the authors have since posted a clarified arXiv version.
  • The most reliable results came from participants actively steering their agents; in the challenge's "human-in-the-loop winner," a person personally judged all 128 image pairs behind a quantization paper's claim because usability was a perceptual call the numerical metrics could not make.

Why it matters

Peer review has not kept pace with the flood of submissions it is partly responsible for. ICML 2026 received 23,918 submissions and accepted 6,352 papers, roughly double the previous year, a jump Hugging Face and alphaXiv tie in part to AI agents making it faster to run experiments and write papers up. Volunteer reviewers cannot fully check thousands of proofs by hand, which is exactly what the post's opening quote, from a reviewer of an ICML 2026 spotlight paper, admits outright. The response here is to turn the same agent capability that is driving submission volume into a scaled, public, post-publication check: an agent that can read a paper, write code, run the experiment and report back in an afternoon, repeated thousands of times over by volunteers instead of conference staff. Hugging Face and alphaXiv call the result probably the largest attempted reproduction of a scientific conference, and the largest open, claim-by-claim audit of a machine learning conference to date.

Who it affects

ICML 2026 authors most directly. The challenge returned verdicts, from fully verified to fully falsified, on a subset of the conference's 6,352 accepted papers. Four papers get named, worked-through treatment: "Towards Optimal Robustness in Learning-Augmented Paging" (an ICML 2026 spotlight paper whose claimed robustness bound did not hold), "Attention's forward pass and Frank-Wolfe" (whose central theorem was disproven by three separate counterexamples, confirmed by its own authors the same day), "Self-Distillation Enables Continual Learning" (whose released code computed the wrong divergence and could not reproduce its own headline number, since corrected on arXiv), and "Do Transformers Need Three Projections?" (whose reported quality cost was inflated by unaccounted padding tokens). Beyond named authors, it affects the 1,221 people who took part as reproducers, and anyone using coding agents for research verification work, since the post is as much about where those agents fail as where they succeed.

How to use it

There is no product or price here. The challenge was free to join, funded by $20 of Hugging Face compute credit per participant, and its output is a public research record rather than a subscription. Every logbook, verdict, trace and artifact from the challenge is public, starting from the challenge's own Space, so any ICML 2026 author or reader can look up how a specific paper fared and read the actual reasoning behind the verdict. The mechanics are reusable too: a streamlined interface let an agent pull a paper, its extracted claims and the challenge instructions with a single command, then run experiments on HF Jobs; that same pattern is what a future participant, or a different conference, could point at a new set of papers. Hugging Face says it plans further reproduction events, without giving a date.

How solid is it

The organizers show real self-skepticism. They say they adversarially re-verified all 35 formally claimed falsifications by re-reading the paper, re-reading the reproduction logbook, and re-deriving the math or re-running the experiment from scratch, and that process is what caught the one false positive detailed in the post, a claimed 2x slower result that was actually the reproducer's own arithmetic bug. Authors of the flagged papers have responded quickly and, so far, in agreement: same-day confirmation in one case, a clarified arXiv upload in another, two further corrections in progress, and one case where the author had already fixed the issue a month before the challenge found it. Set against that, the post's own numbers do not fully reconcile: its title claims 2,200 papers were reproduced, but the four outcome buckets given in the body, 1,103 verified, 496 falsified or contested, 502 toy-scale only, and 280 undetermined, sum to 2,381, and the piece never explains the gap. It also does not state what share of the 6,352 accepted papers were actually examined, or whether the 1,221 participants worked on distinct papers or duplicated each other's picks, and the "largest ever" framing is the organizers' own claim rather than one checked against a documented prior record.

Risks and caveats

The post is explicit that agents alone are not reliable auditors yet. They got stuck in local loops, misread behavior that only shows up at larger scale (several of the paging paper's "verified" verdicts came from checks that stopped before the slower-than-claimed growth became visible), and in one case built a falsification entirely on a plain units mismatch. The clearest illustration is the reverse case in the post itself: a participant's own logbook wrongly claimed a paper's method was twice as slow as the baseline, an error caught only on review; corrected, the paper's original 8x speedup claim held up instead. The organizers' own conclusion is that the most reliable runs had a human actively steering the agent: re-pointing it, questioning an assumption, or stopping a run before it burned a week of compute on a premise that was already wrong. Some judgment stayed irreducibly human in this challenge, too: whether images generated under extreme quantization were actually usable was a perceptual call that numerical metrics could not settle, so a human reviewer judged all 128 image pairs by hand instead.

“My low confidence score is because I did not check all the proofs carefully.”

— an ICML 2026 reviewer, quoted anonymously in the Hugging Face and alphaXiv recap