Thinkingbox benchmark: top AI agent reliable only 25% of the time

Researchers have released Thinkingbox, a sandbox for testing how AI agents handle multi-turn business work, together with Thinkingbox-bench, a set of 507 policy-conditioned tasks built on top of it. The sandbox gives each agent an isolated, MCP-compatible tool session, records the complete execution trace, and grades the outcome by checking the terminal state of a backend system rather than just the agent's final message. The 507 tasks span numerous scenarios, including retail, hospitality, auto insurance, neobank internal IT, and consulting IT and HR support. Each task is judged by a check written for that specific task: it accepts only trajectories that produce the correct effects and rejects ones that get the wrong result, skip a required change, or make an extra change that should not have happened; some tasks also check that the agent's final written response meets required properties. Tested across a mix of proprietary and open-weight models, the strongest model reaches 65.36% pass@1, meaning it completes a given task correctly on a single attempt roughly two out of three times. Under pass^20, a stricter measure of whether the same model can complete the same task correctly on every one of 20 separate attempts, that figure falls to 25.25%. The paper reports that many of the failed attempts do not look like failures on the surface: the agent terminates cleanly and takes state-changing actions that are individually valid, yet the end result is still wrong. The authors argue this shows that signals at the level of the agent's response or its tool calls are not reliable stand-ins for whether the underlying business task was actually completed correctly. Their conclusion is that Thinkingbox-bench exposes a large gap between an agent occasionally finding a successful path through a task and an agent reliably completing that kind of stateful business work every time. Both Thinkingbox and Thinkingbox-bench have been released publicly, with the code posted under Microsoft's GitHub organization at github.com/microsoft/thinkingbox.

Key facts

Thinkingbox-bench contains 507 policy-conditioned workflows spanning retail, hospitality, auto insurance, neobank internal IT, and consulting IT/HR support scenarios.
The strongest model tested reaches 65.36% pass@1 but only 25.25% under pass^20, the metric requiring correct completion across 20 separate attempts at the same task.
Tasks are graded by checks against the terminal backend state: a trajectory is accepted only if it produces the correct effects with nothing wrong, missing, or extra.
Many failed attempts still show clean termination and individually valid state-changing actions, meaning response- and tool-call-level signals do not reliably predict whether the task was actually completed correctly.
Both Thinkingbox and Thinkingbox-bench are released publicly, with the code hosted under Microsoft's GitHub organization.

Why it matters

Most agent benchmarks score whether a single response or tool call looks right. Thinkingbox instead checks what actually happened to a backend system after a multi-turn interaction, and that shift changes the picture: an agent that looks like it finished the job cleanly can still have left the wrong record behind. For work that changes real state, such as an insurance claim, a retail return, or an IT ticket, a single clean-looking run is not proof the task was done correctly, and a model that succeeds two times out of three on a first try is not the same as one that can be trusted to do so every time.

Who it affects

Anyone building or evaluating agents meant to carry out business processes rather than answer questions, including teams working on retail and hospitality support automation, auto insurance claims handling, internal IT support at banks, and consulting IT or HR helpdesks. It also matters to benchmark designers and researchers studying agent reliability, since the paper argues that existing response- and tool-call-level metrics are not good substitutes for checking the actual outcome.

How to use it

Thinkingbox and Thinkingbox-bench are both released publicly, with the code posted at github.com/microsoft/thinkingbox. Teams can run their own agents against the 507 workflows through the sandbox's isolated MCP-compatible tool sessions and evaluate them against the same terminal-state checks used in the paper, rather than relying on the agent's own report of success.

How solid is it

The evaluation method is built around task-specific executable checks that inspect the backend state after each run, accepting only trajectories with the correct effects and rejecting ones with wrong, missing, or extra changes, which is a stricter bar than judging a transcript by eye. The paper tested a mix of proprietary and open-weight models, though the source text does not name which specific models were evaluated or which one produced the 65.36% and 25.25% figures, and it does not describe the executable checks in more technical detail than that. It also does not state the authors' names or institutional affiliation, beyond the fact that the released code sits under Microsoft's GitHub organization.

Risks and caveats

Because the source does not name the specific models tested, it is not possible to say from this text alone which system achieved the top pass@1 and pass^20 scores, or how far behind the other tested models fell. The pass^20 metric and the 507 tasks are the paper's own design, and how well they predict reliability in a different company's actual workflows has not been independently confirmed. As with any newly introduced benchmark, the numbers describe performance on this specific test suite rather than a general guarantee about agent behavior elsewhere.

“many failed trials show clean termination and valid state-changing actions, showing that response or tool-call-level signals are not clear proxies for end-to-end task completion”

— Thinkingbox-bench paper