METR updates its AI time horizon plot with harder, open-ended tasks

The AI evaluation nonprofit METR has published an update to its time horizon plot, one of the most widely shared and, according to MIT Technology Review, most widely misinterpreted charts in AI. The new report finds that some AI capabilities are still improving along an exponential trajectory, but that models are not quite as good at tasks requiring judgment as they are at coding.
The original method had models attempt a large suite of tasks that humans had also completed. Each task's difficulty was scored by how long it took the human testers, and models were rated by the length of the tasks they could complete. That plot showed the length of tasks frontier models could successfully complete doubling every four months.
The updated plot adds more tasks: some estimated to take humans far longer than those in the original suite, and others that are more open-ended. This matters because of a central open question: whether progress on "verifiable" tasks, where software can check automatically whether the AI succeeded, carries over to tasks where success is harder to measure. Reinforcement learning, the technique behind recent gains in coding and math, works well where success can be checked automatically. It is much harder to apply to tasks such as choosing a new research direction, and those tasks are also hard to evaluate, so the original plot included none of them. They are the kind of task that would matter if AI is to automate research and development.
The topic is especially live because of a different, heavily redacted METR report from earlier this month. It claimed AI might be speeding up AI progress at Anthropic by a factor of two, and sparked fears that the company, which just had a blockbuster IPO, could soon reach full recursive self-improvement, where AI systems design their own successors with no human involvement.
METR's new suite includes some fuzzier, judgment-laden tasks. In one, models design new AI evaluations that METR could use in its research, and human METR employees score the results. Even though the most advanced models are now better than almost every human at software engineering, here they fall short: not one of the AI-designed evaluations was good enough for METR employees to consider it useful for their research. In another task, models score philosophical arguments, which is meant to capture how well they exercise judgment. METR writes in the report that employees disagree about how relevant this task is for automating AI R&D. Models fell short of human performance here too.
Experts the article's author spoke with welcome the direction. Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley, says evaluations need to get more creative. He was not surprised by the shortfall on open-ended tasks: in an August preprint, he and his collaborators reported that AI models cannot yet reproduce human computer science research on their own. In a Wednesday blog post, OpenAI reported that its models do not do as well on the real-world tasks its employees assign them as on the easier, more verifiable tasks in METR's original suite. Kapoor thinks research taste, the ability to choose promising questions, hypotheses and experiments, will prove a major bottleneck in automating AI R&D. Others contend that models trained on easy-to-verify tasks will eventually complete open-ended tasks too, and progress on verifiable tasks does not seem to be slowing down.
There are signs of movement. Some of the newest models, such as Opus 5.5, tend to reach what METR calls "borderline above the bar" on the evaluation design tasks: not yet usable for METR's research, but promising. A year ago, every model scored "clearly below the bar".
But METR includes few open-ended, judgment-heavy tasks because they are so hard to design and score, which makes it difficult to predict how models will progress in these areas. Najoung Kim, a linguist and computer scientist at Boston University, says the observed improvement tells us something for the current moment but she is not sure yet how it would extrapolate. Matt Putz, a member of the technical staff at METR, says in an email that hard-to-verify tasks are much harder to study and quantify, and that he does not expect trends as clean and robust as for easy-to-specify tasks.
The practical limits are real. For the original plot, METR measured human completion times by having people try each task. In the new report the times are mostly estimates, because paying people for tasks that might take hundreds of hours, or that only a few experts can do, is prohibitively expensive and time-consuming. For the philosophical argument task, just one person rated most of the arguments, and agents are judged by how well they agree with that individual; scores might change if more experts were surveyed, which is also time-consuming. The experts see the inclusion of these tasks as a clear step in the right direction, but agree we are far from definitive answers on whether, and how quickly, automated AI R&D might arrive.
Key facts
- METR's updated time horizon plot adds tasks estimated to take humans far longer and tasks that are more open-ended and judgment-heavy; the original plot showed frontier models' completable task length doubling every four months.
- In the evaluation-design task, not one AI-designed evaluation was good enough for METR employees to consider it useful for their research; models also fell short of humans at scoring philosophical arguments.
- Some of the newest models, such as Opus 5.5, tend to reach METR's "borderline above the bar" on evaluation design; a year ago every model scored "clearly below the bar".
- Human completion times in the new report are mostly estimates, and one person rated most of the philosophical arguments, so the new tasks are less firmly measured.
- METR's Matt Putz does not expect trends as clean and robust for hard-to-verify tasks as for easy-to-specify ones.
Why it matters
The time horizon plot is a reference point in debates about how fast AI is improving. Until now it covered only tasks with automatic success checks, the area where reinforcement learning works best. The update is a first attempt to test the harder question: whether those gains carry into judgment-heavy work such as choosing research directions, which would matter if AI is to automate AI research and development. The timing is charged, since an earlier, heavily redacted METR report this month claimed AI might be speeding up AI progress at Anthropic by a factor of two and set off fears of recursive self-improvement.
Who it affects
AI researchers and evaluators, who are being pushed toward more creative and open-ended benchmarks. AI labs and anyone following claims about automated AI R&D, including the debate around Anthropic. Readers who see the time horizon chart cited in discussions of AI progress.
How to use it
This is a research report, not a product, so there is nothing to adopt. The practical use is as a reading guide. When you see a time-horizon chart, check what kind of tasks sit behind it: verifiable ones like coding and math, or open-ended and judgment-heavy ones. Also check whether human completion times were measured or estimated. Kapoor's view that research taste may be a bottleneck, and the opposing view that training on easy-to-verify tasks will eventually carry over, are the two positions to weigh.
How solid is it
The reporting comes from MIT Technology Review, which describes METR's report and quotes outside experts and a METR staff member. The exponential finding applies to some capabilities, mainly the verifiable ones, and progress there does not seem to be slowing. The judgment-heavy results rest on few tasks. Human completion times are mostly estimates, and one person rated most of the philosophical arguments. METR itself says employees disagree about how relevant that task is for automating AI R&D. Kim says she is not sure how the improvement would extrapolate.
Risks and caveats
Few judgment-heavy tasks are in the report, so it is hard to predict whether and how models will progress there, and Putz does not expect trends as clean as for easy-to-specify tasks. The shortfall on evaluation design is a snapshot: newer models such as Opus 5.5 are nearing "borderline above the bar", so the picture may shift. The earlier Anthropic speed-up claim is only that AI might be doing this, and the report was heavily redacted. The experts agree that we are still far from definitive answers about whether, and how quickly, automated AI R&D might arrive.
“We’re at the point where AI evaluations need to get more creative”
— Sayash Kapoor, a computer scientist and incoming professor at UC Berkeley