ASI-Bench shows AI research agents falter without human guidance
Researchers have introduced ASI-Bench, a benchmark built to measure whether AI systems can go beyond applying existing knowledge and actually explore the unknown, generate new ideas, and turn those ideas into verifiable scientific results. The team behind it argues that most existing AI evaluations only test whether a system can produce a correct answer from what it already knows, or complete a task under heavy human direction. ASI-Bench is described as the first benchmark to jointly evaluate innovative exploration and autonomous scientific execution across general research domains, and the first to progressively withdraw human methodological guidance within the same research project, to see how far an AI system can get on its own.
The benchmark was built by over 40 experts at a cost of more than 31,000 human hours. It contains 60 project-level research tasks spanning 11 scientific domains. Guidance is dialed back in stages within the same task: full methodological guidance, guidance that specifies only the method to use, and no guidance at all, in which case the AI must determine its own method. Every task goes through expert review, AI-assisted auditing, sandbox execution, and scorer validation before results count.
The authors tested 18 state-of-the-art agent-model configurations across these three guidance levels. The average score was 50.91 with full methodological guidance, fell to 29.10 when only the method was specified, and dropped further to 26.62 when agents had to determine the method themselves. The authors say this sharp decline shows that current AI systems remain heavily dependent on human guidance and are still far from autonomously conducting end-to-end, project-level scientific research. ASI-Bench is open for outside contribution: researchers and builders are invited to submit new tasks through the project's website.
Key facts
- ASI-Bench is billed as the first benchmark to jointly test AI's innovative exploration and autonomous scientific execution, and the first to progressively withdraw human guidance within one research project.
- It was built by over 40 experts at a cost of 31,000+ human hours, and contains 60 project-level research tasks across 11 scientific domains.
- Across 18 state-of-the-art agent-model configurations, the average score fell from 50.91 with full methodological guidance to 29.10 when only the method was specified.
- With no guidance at all, agents had to choose their own method and the average score dropped further, to 26.62.
- Every task passes through expert review, AI-assisted auditing, sandbox execution, and scorer validation, and the benchmark is open for outside teams to contribute new tasks.
Why it matters
Most AI benchmarks measure whether a system can recall or apply knowledge it was trained on, or complete a task while a human supplies the method. ASI-Bench targets a different question: what happens when that human methodological scaffolding is taken away, task by task, within the same project. The result is one of the more direct attempts yet to quantify the gap between AI that assists research and AI that could independently drive it, which is the capability the benchmark's name points toward.
Who it affects
The benchmark is aimed at teams building and evaluating AI research agents, and at labs claiming autonomous scientific capability for their models. The 18 agent-model configurations tested span current state-of-the-art systems, so the results speak to the field broadly rather than to one vendor.
How to use it
ASI-Bench is open: researchers and builders can submit new tasks through the project's site at asibench.apexin.ai/submit. The source gives no pricing, licence terms, or access restrictions for using the benchmark itself.
How solid is it
The benchmark rests on a substantial build effort: over 40 experts, more than 31,000 human hours, 60 tasks across 11 domains, and 18 tested agent-model configurations. Every task is checked through expert review, AI-assisted auditing, sandbox execution, and scorer validation before a score counts, which is a heavier verification pipeline than most benchmarks disclose.
Risks and caveats
The source text names no authors or institutions, gives no publication or submission date, and does not break down which of the 18 agent-model configurations scored highest or lowest. It also does not describe what the 11 domains or 60 individual tasks actually involve, or explain how the scorer validation and AI-assisted auditing work in practice. As with any benchmark introduced by its own creators, the claims about what it measures have not yet been independently tested by outside teams.
“This sharp decline shows that current systems remain heavily dependent on human guidance and are still far from autonomously conducting end-to-end, project-level scientific research.”
— ASI-Bench authors