Glonce

Agent4Sci tests coding agents on 20 science ML benchmarks

researchThe Glonce editorsHugging Face🔥59Glonce rating59Impact45Novelty55Relevance100Surprise30Players30Credibility70
Agent4Sci tests coding agents on 20 science ML benchmarks

Frontier coding agents are increasingly framed as end-to-end automated researchers. A new paper tests that framing on machine learning for science, where improving a model often takes domain insight beyond routine ML engineering. The authors introduce Agent4Sci, a long-horizon testbed that measures how far today's agents can improve the published best models on scientific ML benchmarks. It spans 20 benchmarks across 11 domains.

The authors say Agent4Sci is not meant as a leaderboard. Instead it asks what kinds of progress agents can make on these tasks under in-the-wild conditions, and where they fall short.

The headline result is mixed. Today's frontier agents advance the published best on every benchmark, with Claude Opus reaching 10.4% mean relative improvement, and no run was flagged for academic misconduct. But the authors write that these gains come almost entirely from a bag of ML-engineering tricks rather than research-level insight.

Trajectory analysis of the agents' runs surfaces three recurring limitations: novelty miscalibration, resource under-utilization, and rapid collapse into local engineering sweeps.

The authors then add lightweight plug-in skills for research ideation, experiment management, and scheduled checklists. With them, Claude Opus rises from 10.4% to 20.1% and Codex from 4.4% to 8.1%. The authors also say the skills yield more diverse, domain-relevant interventions.

The authors position Agent4Sci as a way for the community to track and push frontier agents from ML engineering toward domain insight on real-world ML-for-science problems. Code is on GitHub (github.com/maszhongming/agent4sci) and data is on Hugging Face (huggingface.co/datasets/MingZhong/agent4sci).

Key facts

  • Agent4Sci is a long-horizon testbed covering 20 scientific ML benchmarks across 11 domains, built to measure how far agents can improve the published best models.
  • Frontier agents advance the published best on every benchmark; Claude Opus reaches 10.4% mean relative improvement, and no run was flagged for academic misconduct.
  • The authors say the gains come almost entirely from a bag of ML-engineering tricks rather than research-level insight.
  • Trajectory analysis finds three recurring limitations: novelty miscalibration, resource under-utilization, and rapid collapse into local engineering sweeps.
  • Plug-in skills for ideation, experiment management and scheduled checklists raise Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%.

Why it matters

Agents are often described as end-to-end automated researchers. Agent4Sci tests that claim where it should be hardest, in scientific ML, where progress often needs domain insight rather than routine engineering. The answer in the abstract is that agents do beat the published best on every benchmark, yet the authors attribute the gains almost entirely to ML-engineering tricks, not research-level insight. The testbed is framed as a way to track movement from engineering toward domain insight, not as a leaderboard.

Who it affects

Researchers building or evaluating research agents get a testbed and a list of failure patterns: novelty miscalibration, resource under-utilization, and collapse into local engineering sweeps. Scientists who work with ML in their own domains may care about the finding that agents can improve published best models, though the authors stress the improvements are mostly engineering ones. The paper evaluates Claude Opus and Codex.

How to use it

Code is on GitHub at github.com/maszhongming/agent4sci and data is on Hugging Face at huggingface.co/datasets/MingZhong/agent4sci. The paper also describes lightweight plug-in skills for research ideation, experiment management, and scheduled checklists. In the authors' tests they raised Claude Opus from 10.4% to 20.1% and Codex from 4.4% to 8.1%, and produced more diverse, domain-relevant interventions.

How solid is it

This is a paper abstract, so the claims are the authors' own summary of their results. The headline figure is Claude Opus at 10.4% mean relative improvement over the published best, across 20 benchmarks in 11 domains. The source does not say which metric the 20.1%, 4.4% and 8.1% figures use. No specific Claude Opus or Codex model versions are given. No per-benchmark or per-domain results are given. No run length, compute budget or number of runs is given. The abstract does not say how misconduct was checked.

Risks and caveats

The authors' own conclusion is a caveat: the improvements are largely engineering tricks, so they should not be read as evidence of agents producing scientific insight. The plug-in skills roughly double the reported gains, but the abstract gives no further detail on how robust that is. Only Claude Opus and Codex are named; the abstract does not say whether other agents were tested. Agent4Sci is intended to show where agents fall short, not to rank them.

“These gains, however, come almost entirely from a bag of ML-engineering tricks rather than research-level insight.”

— Agent4Sci paper abstract