Glonce

SWE-bench Science finds top coding agent scores below 50% on science tasks

researchThe Glonce editorsHugging Face🔥56Glonce rating56Impact60Novelty75Relevance95Surprise55Players50Credibility85
SWE-bench Science finds top coding agent scores below 50% on science tasks

Researchers have introduced SWE-bench Science, a repository-level benchmark built to test whether coding agents can actually fix scientific software, not just generic code. The benchmark comprises 119 tasks drawn from 98 GitHub repositories spanning 20 scientific domains, and each task falls into one of three paradigms: Issue-driven, Expert-exploratory, and Engineering-integration. The motivation is that software has become part of the scientific instrument itself, so a bug in scientific code can corrupt not just program behavior but the evidence behind a scientific conclusion, yet most existing coding-agent evaluations only track aggregate pass rates and say little about why agents fail on this kind of work. On SWE-bench Science, even the best-performing agent the authors tested, Claude Code with Opus-5 (max), achieves a pass@1 score below 50%. Analyzing the failures, the authors identify four recurring failure mechanisms: gaps in scientific knowledge or abstraction, misguided exploration or surface-level repairs that do not address the underlying issue, incomplete repair coverage or system integration, and an inability to generalize scientific knowledge beyond the specific cases the agent has already seen. The authors also ran a paired ablation that strips out explicit scientific guidance while keeping the repository and executable engineering context intact. That test shows scientific knowledge is not uniformly helpful: guidance that is well-grounded can constrain the agent's repairs in a useful way, improving both average performance and token efficiency, but guidance that is poorly aligned with the actual problem can cause the agent to anchor on the wrong fix and does not reliably improve the chance of an exact, correct repair.

Key facts

  • SWE-bench Science is a new benchmark of 119 tasks from 98 GitHub repositories across 20 scientific domains, split into Issue-driven, Expert-exploratory, and Engineering-integration task types.
  • The best agent tested, Claude Code with Opus-5 (max), still scores a pass@1 below 50% on the benchmark.
  • The authors identify four recurring failure mechanisms: scientific-knowledge or abstraction gaps, misguided or surface-level repairs, incomplete repair coverage, and failure to generalize knowledge to new cases.
  • A paired ablation shows scientific guidance is not uniformly beneficial: well-grounded guidance improves average performance and token efficiency, while poorly aligned guidance can anchor the agent on the wrong fix without improving exact repair success.

Why it matters

Coding agents are increasingly pitched as tools for research work, but most benchmarks measure them on general software tasks where a bug just breaks a feature. Scientific software is different: a repair mistake can silently corrupt the evidence behind a published finding, not just crash a program. SWE-bench Science is built specifically to probe that gap, and the headline result, a sub-50% pass@1 for the strongest agent tested, shows current agents are still far from reliable on this class of work.

Who it affects

The benchmark is aimed at anyone building or evaluating coding agents for research and engineering use, and at scientists and labs who might otherwise assume an agent can safely maintain or repair the software behind their results. It is also a direct data point for teams building agents like Claude Code, since one such agent, Claude Code with Opus-5 (max), is the top performer in the study and still falls short of even coin-flip reliability.

How to use it

SWE-bench Science functions as a benchmark suite: it groups its 119 tasks into three paradigms, Issue-driven, Expert-exploratory, and Engineering-integration, spanning 20 scientific domains from 98 real GitHub repositories, so it can be used to evaluate a coding agent's repair ability against realistic scientific codebases rather than synthetic problems.

How solid is it

The benchmark rests on real repositories and tasks across a broad domain spread, 98 repositories and 20 domains, which lends it practical grounding, and the failure analysis is based on a paired ablation methodology, comparing agent performance with and without explicit scientific guidance while holding the repository and engineering context fixed. The abstract does not give an exact pass@1 figure for the top agent, only that it is below 50%, and it does not name the four failure mechanisms with specific example tasks in the summarized text.

Risks and caveats

The finding that scientific guidance is not uniformly beneficial cuts against a simple assumption that more domain knowledge always helps an agent: poorly aligned guidance can cause anchoring, pulling the agent toward a plausible but wrong fix, and does not reliably raise the odds of an exact correct repair. That nuance matters for anyone designing prompts or retrieval systems meant to feed scientific context to a coding agent, since adding guidance without checking its alignment to the actual problem can backfire.