New benchmark: Claude Opus 5 tops out at 23.9% on building real agents

Researchers introduced tau-tau-bench, pronounced hyper-tau-bench, a benchmark built around a different question than existing agent tests ask: not whether a model can answer well inside a fixed agent, but whether a coding agent can build one that works under the conditions of a real client engagement. Each task hands a developer agent the same starting point a real project would: the records a business actually keeps, a client who holds the requirements, a production API that operations has to run through, an existing codebase to inherit, and limits on serving cost and which models can be used. From that, the developer agent has to deliver a complete customer-service agent. Scoring does not stop at code review: the finished agent is deployed and run against held-out simulated users, so the benchmark measures whether the thing actually works in use, not just whether it compiles or looks reasonable. The benchmark spans 53 tasks across four domains. Across every configuration tested, the strongest was Claude Opus 5 running under Claude Code, and it passed just 23.9% of the evaluation simulations. An expert-authored reference solution, built by a human for the same tasks, scores 82.2%, roughly 3.4 times the best AI result. The researchers describe the failure modes as mirroring what human agent developers see on the job: the AI systems issue shallow queries against the records instead of building deep comprehension of them, communicate almost nothing back to the client during the build, and do too little experimentation with agent architecture or serving spend, typically shipping the first design that happens to run rather than iterating toward a better one. The stated goal is to turn the cooperative, back-and-forth work of building an agent for a real business into a measurable target that coding agents can be trained and evaluated against, rather than leaving that gap invisible to existing benchmarks.

Key facts

tau-tau-bench scores a developer agent on building a full customer-service agent from a business's records, a client's requirements, a production API, an inherited codebase, and cost and model limits, then deploying it against held-out simulated users.
The benchmark covers 53 tasks across four domains.
The strongest configuration tested, Claude Opus 5 under Claude Code, passed only 23.9% of evaluation simulations.
An expert-authored reference solution scores 82.2%, about 3.4 times the best AI result.
Failure modes include shallow queries instead of deep understanding of the records, near-silence toward the client during the build, and shipping the first working design instead of iterating on architecture or cost.

Why it matters

Most agent benchmarks test a fixed agent against fixed tasks. tau-tau-bench tests something closer to the actual job now being handed to coding agents: building the agent itself, from a real business's records and a client's requirements, under real constraints like a production API and a serving-cost ceiling. That is the task companies are already delegating to AI coding tools, and this benchmark is one of the first to measure how well that delegation actually works end to end, by running the finished product against simulated users rather than just reviewing the code.

Who it affects

Teams building or buying coding agents to construct production customer-service systems, and any business considering handing that construction work to an AI system rather than a human developer or agency. It also matters to researchers building agent benchmarks, since tau-tau-bench targets a gap the authors say existing benchmarks say little about.

How to use it

tau-tau-bench is a benchmark, not a product or service with a price or license. Its practical use is as a yardstick: an organization evaluating whether a coding agent is ready to build a customer-facing agent can check how that agent's underlying model performs on these 53 tasks, and weigh a 23.9% pass rate for the best current configuration against the 82.2% an expert developer reaches on the same tasks.

How solid is it

The source is a paper page with a full-length abstract describing the benchmark's design, task count, domain count, and headline results, but it carries no author names, institutions, or affiliations in the text, and no comparison to any specific prior benchmark. It has one point and one comment on the platform where it surfaced, so there is little external engagement yet to corroborate the reported figures. The 23.9% and 82.2% figures are stated without a breakdown by domain or task.

Risks and caveats

The gap between 23.9% and 82.2% is large enough that it should be read as an early, unflattering baseline rather than a settled verdict on any model's agent-building ability. The text does not name the specific four domains the 53 tasks span, does not give a release date or availability for the benchmark, and does not break the headline numbers down further, so readers cannot yet see whether the failures cluster in particular domains or task types.

“models issue shallow queries in place of deep comprehension of the records, communicate almost nothing to the client, and experiment too little with agent architecture and serving spend, shipping the first design that runs”

— tau-tau-bench paper