Glonce

Terminal-Bench-Science debuts: Claude Opus 5 resolves 30% of tasks

researchThe Glonce editorsHacker News🔥74Glonce rating74Impact65Novelty78Relevance95Surprise50Players75Credibility85

Terminal-Bench-Science 0.1 is a new benchmark for evaluating AI agents on real scientific research workflows, built by researchers at Stanford University together with the team behind Terminal-Bench and domain experts from research institutions worldwide. Rather than textbook questions, its 70 tasks come from practicing scientists' own work, spanning the life, physical, Earth, mathematical, and engineering sciences. The stated goal is to measure whether AI agents can take on technically demanding, time-consuming research work such as data analysis, simulation, and proof, freeing scientists to focus on defining questions, forming hypotheses, and interpreting results.

The 70 tasks were filtered from a much larger pool. Researchers submitted 920 task proposals through an open process on GitHub, with discussion in a Discord channel; 464 were approved for implementation and turned into 386 pull requests, but only 70 made it into the 0.1 release. Each surviving task had to pass review from domain reviewers checking scientific validity and realism, technical reviewers checking task construction and verification, and a bar raiser doing a final quality check, confirming the task is objectively verifiable and genuinely hard for today's frontier agents rather than something they already solve easily.

Each model ran three independent trials per task across all 70 tasks. Claude Opus 5, running with Claude Code, came out on top with a 30.0% resolution rate, ahead of GPT-5.6 Sol running with Codex at 22.4% and Claude Fable 5 running with Claude Code at 21.4%. Claude Opus 4.8 sat in the middle at 10.5%. GPT-5.6 Terra, Kimi K3, and Grok 4.6 all resolved less than 10% of tasks. GLM 5.3 was the strongest open-weight model at 8.1%, and GPT-5.6 Luna finished last at 3.3%. Every model that had also run on Terminal-Bench 3.0 scored more than 10 percentage points lower here, a gap the Terminal-Bench-Science team says is deliberate: tasks were calibrated during review specifically to challenge the newest frontier models, while the benchmark still separates strong from weak systems about as well as Terminal-Bench 3.0 does.

Resolution rate was not the only axis measured. Across all 70 tasks, Claude Opus 5's run cost a total of $7.0k, well above the frontier's low-cost end. GPT-5.6 Sol matched Claude Fable 5's resolution rate for less than a third of the cost ($4.2k versus $14.2k), and Claude Fable 5 matched GPT-5.6 Sol's rate using about a quarter fewer tokens (6.4 billion versus 8.4 billion). By domain, Anthropic's and OpenAI's models took the top two spots almost everywhere: Claude Opus 5 led every domain except the mathematical sciences, where Claude Fable 5 (33.3%) and GPT-5.6 Sol (31.4%) took the top two spots instead. The exception to that sweep was the engineering sciences, where Grok 4.6 tied GPT-5.6 Sol for second place at 14.8%, spending less on compute and fewer tokens to get there.

Terminal-Bench-Science is designed to be a continuous benchmark rather than a one-off paper: tasks are versioned so trials can be reused, re-graded, or rerun with a single Harbor command, and progress is tracked openly on a public task dashboard. Work on the next release, Terminal-Bench-Science 0.2, is already underway, with a pull request deadline of October 5, 2026; researchers can contribute a task through a proposal form, build it following a public contributing guide, and take it through automated checks plus the same domain, technical, and bar-raiser review before it merges. The project is hosted by Stanford University and the Laude Institute, in partnership with the Stanford AI Lab, the Stanford Institute for Human-Centered Artificial Intelligence, Stanford AI Measurement Science, the NSF AI Institute for Foundations of Machine Learning, the Allen Institute, and the Allen Institute for AI (Ai2), with Ludwig Schmidt and Sanmi Koyejo credited as project lead advisors. The announcement was written by Steven Dillmann.

Key facts

  • Terminal-Bench-Science 0.1, led by Stanford researchers and built by the Terminal-Bench team with domain experts worldwide, tests AI agents on 70 real research tasks across five scientific domains: life, physical, Earth, mathematical, and engineering.
  • The 70 tasks survived a funnel of 920 proposals, 464 approvals, and 386 pull requests, each vetted by domain reviewers, technical reviewers, and a bar raiser.
  • Claude Opus 5 with Claude Code resolves the most tasks at 30.0%, ahead of GPT-5.6 Sol with Codex (22.4%) and Claude Fable 5 with Claude Code (21.4%); GLM 5.3 is the strongest open-weight model at 8.1%.
  • Every model tested on both benchmarks scores more than 10 percentage points lower here than on Terminal-Bench 3.0, which the team says reflects deliberate calibration against the newest frontier models.
  • The next release, Terminal-Bench-Science 0.2, is already underway with a pull request deadline of October 5, 2026, continuing the benchmark's open, continuous-contribution model.

Why it matters

Most AI benchmarks for science are built by AI labs or data vendors and then published once, left behind as models improve. Terminal-Bench-Science is designed differently: its tasks come from practicing scientists' own workflows, they are reviewed by scientists rather than only by AI researchers, and the benchmark is meant to keep releasing new tasks as frontier models catch up to old ones. The premise is that if AI agents are going to take over technically demanding research labor such as data analysis, simulation, and proof, freeing scientists to spend more time on hypotheses and interpretation, the benchmark measuring that capability needs to reflect what scientists actually do, not textbook exercises. The result on this first release is sobering for anyone expecting agents to already be capable research assistants: even the strongest model resolves under a third of the tasks (30.0%), and every model also measured on the general-purpose Terminal-Bench 3.0 scores more than 10 percentage points lower here, which the team frames as proof the new tasks are genuinely harder rather than just differently scored.

Who it affects

The two organisations behind the top-scoring agents, Anthropic and OpenAI, took the top two spots in nearly every scientific domain: Claude Opus 5 led overall and in every domain except the mathematical sciences, where Claude Fable 5 and GPT-5.6 Sol took first and second. That leaves scientists and research institutions deciding whether current AI agents are ready for real research workflows with a concrete, if unflattering, number to weigh. It also affects the developers of Grok 4.6, Kimi K3, GLM 5.3, and the other tested models, all still resolving less than 10% of tasks and now facing a public leaderboard tracking the gap. The benchmark itself is a joint effort of Stanford University, the Laude Institute, the Stanford AI Lab, the Stanford Institute for Human-Centered Artificial Intelligence, Stanford AI Measurement Science, the NSF AI Institute for Foundations of Machine Learning, the Allen Institute, and the Allen Institute for AI (Ai2), alongside the wider community of scientific task contributors.

How to use it

Terminal-Bench-Science is open for scientists to contribute to, not just read. New tasks go through a Propose, Build, Review pipeline: a researcher submits a task proposal through a form, builds it following a public contributing guide, and then goes through automated checks plus parallel domain and technical review before a bar raiser gives final approval. Coordination happens in the #tb-science channel on Discord and in weekly meetings and office hours listed on the project calendar. Tasks are versioned, so existing trials can be reused, re-graded, or rerun with a single Harbor command instead of being rerun from scratch every release, and progress on proposals, pull requests, and reviews is tracked publicly on a task dashboard. The next release, Terminal-Bench-Science 0.2, already has an open pull request deadline of October 5, 2026.

How solid is it

The review funnel is heavy: of 920 proposals, only 464 were approved for implementation, only 386 of those turned into pull requests, and only 70 survived into the release. Each surviving task passed domain reviewers checking scientific validity and realism, technical reviewers checking construction and verification, and a final bar-raiser quality check confirming the task is objectively verifiable and still hard for today's frontier agents. Grading itself uses reproducible, task-specific tests against concrete artifacts such as analyses, simulations, proofs, code, and data products, rather than one model judging another's answer. Each model ran three independent trials per task across all 70 tasks. The announcement is self-published by the benchmark's own team and credited to writer Steven Dillmann, so these results have not yet been independently reproduced outside the project.

Risks and caveats

The text never states who builds Grok 4.6, Kimi K3, or GLM 5.3; it ties Claude-branded models to Anthropic and GPT-5.6-branded models to OpenAI only through one aggregate sentence about domain rankings. No individual cost or token figures are given for GPT-5.6 Terra, Kimi K3, or Grok 4.6 in the overall leaderboard: GPT-5.6 Luna, Kimi K3, and GPT-5.6 Terra are placed only qualitatively at the low-cost end of the cost-resolution frontier, and GPT-5.6 Terra, Kimi K3, and Grok 4.6 are said only to resolve less than 10% of tasks, with no individual overall rate given for any of the three. No release date is given for Terminal-Bench-Science 0.1 itself, only the October 5, 2026 pull request deadline for the follow-up release, 0.2. The announcement also does not say how the published resolution rate is derived from the three independent trials run per task, whether it is a mean, a median, or a task counted as resolved if any single trial succeeds. And while the acknowledgements thank the Laude Institute, Snorkel AI, the 2077AI Open Source Foundation, UniPat AI, Modal, and API credits from Bespoke Labs, Anthropic, Google, Moonshot AI, SpaceXAI, and Z.ai, no dollar figures are given for any of that support.

“Scientists, not model developers or data vendors, set the bar for scientific capability in AI.”

— Terminal-Bench-Science team