OpenDS-World benchmark: best AI model hits 39% on live-web deep search

A new paper argues that deep search agents, despite recent progress, have not been properly tested on end-to-end research in the real, open web. Unlike static retrieval settings, the live web spreads information across diverse sources, modalities and heterogeneous formats, and often involves complex interaction mechanisms. Agents must search for, access and integrate evidence on their own while they run.

To fill that gap, the paper introduces the OpenDS task. It asks an agent to explore the real web autonomously, find and connect evidence from multiple independent sources using web search, browser interaction and evidence processing, and finally settle on a single unique factual answer.

Building on that task, the authors construct OpenDS-World, a benchmark of 200 challenging tasks. The tasks span diverse information modalities, heterogeneous file formats and interactive access requirements. Independent human verification is used to make sure the evidence is valid and each answer is unique.

The authors evaluate 12 frontier models under a unified tool environment and execution protocol. The best-performing model reaches 39.0% accuracy. Human evaluation across 1,000 independent attempts yields 21.1% accuracy.

Further analyses cover tool usage, search cost, termination behavior and failure recovery. They show substantial differences in research strategies across models, as well as persistent challenges in tool coordination and long-horizon search management. The authors present OpenDS-World as a unified evaluation platform for assessing and advancing agents' autonomous deep search in real-world open web environments.

Key facts

The paper introduces the OpenDS task: multi-step deep search on the live, open web that ends in a single unique factual answer.
OpenDS-World is a benchmark of 200 challenging tasks across diverse modalities, heterogeneous file formats and interactive access requirements, with independent human verification.
12 frontier models were tested under a unified tool environment and execution protocol; the best reached 39.0% accuracy.
Human evaluation across 1,000 independent attempts yielded 21.1% accuracy.
Analyses of tool usage, search cost, termination behavior and failure recovery show large differences in research strategies and persistent trouble with tool coordination and long-horizon search management.

Why it matters

The authors say agents' ability to do end-to-end research on the real open web is still insufficiently evaluated. Static retrieval benchmarks do not capture a web where evidence sits in varied formats and behind interactive access. OpenDS-World is built to test exactly that, with tasks that need evidence from several independent sources and end in one verifiable answer.

Who it affects

Teams building or evaluating deep search and web-browsing agents are the direct audience, since the benchmark is meant as a shared evaluation platform. Developers of frontier models are also affected, because 12 of them were run through the same tool environment and protocol.

How to use it

The abstract describes OpenDS-World as a unified platform for systematically assessing and advancing agents' autonomous deep search. Its analyses of tool usage, search cost, termination behavior and failure recovery point to the areas to inspect when comparing agents: how they coordinate tools and manage long searches.

How solid is it

The benchmark is small, at 200 tasks, but each task was independently verified by humans for evidence validity and answer uniqueness. All 12 models ran under one unified tool environment and execution protocol. The 39.0% best-model score and the 21.1% human score come from the paper's own evaluation. The names of the models are not given in the text, and neither are per-model results beyond the best one. The text does not say whether the model and human figures were measured on the same tasks or under the same conditions, so they should not be read as a head-to-head comparison.

Risks and caveats

The 21.1% figure comes from 1,000 independent human attempts; the number of participants is not stated. The text does not say which model scored 39.0%, and it mentions no code or dataset release, link or license. The findings rest on a 200-task set evaluated on the live web, and the paper itself points to persistent challenges in tool coordination and long-horizon search management.