Glonce

Taste-Bench: best of 14 AI models scores 59.7% on long-term agent decisions

researchThe Glonce editorsThe Decoder🔥57Glonce rating57Impact40Novelty55Relevance95Surprise35Players40Credibility65
Taste-Bench: best of 14 AI models scores 59.7% on long-term agent decisions

Researchers from Microsoft and City University of Hong Kong have built Taste-Bench, a benchmark for the quality of the decisions AI agents make during long, multi-step tasks such as writing a research paper or building a software project. Along the way an agent keeps choosing which hypothesis to test, which code to build on, or which experiment to run next. Those choices rarely show their effects right away: a bad decision often looks fine at first, and its true cost appears only after most of the agent's budget is spent. The paper's authors call the ability to choose well an agent's "taste". Existing benchmarks only measure whether an agent finishes a task, not the quality of its steps along the way.

The difficulty is that a long-term choice cannot be judged in the moment. A clean implementation and a sloppy one may give the same output and pass the same tests; the clean one pays off only when later changes come more easily. Human grading would be expensive, need deep domain expertise and not scale. So the researchers rely on hindsight. When an agent system tries the same task several times, the attempts eventually split in different directions, and the final result of each attempt shows which path was better. The authors call this branching point a "decision fork". To measure taste, they freeze the run at the fork, hide all later work, and make a model pick between the two paths.

Taste-Bench contains 502 questions. Each has the original task, the trajectory up to the fork, and two possible directions. One type comes from two independent attempts at the same task that diverge, where the agent never sees the wrong path. The other type comes from an agent correcting itself within a single run and taking a detour: the initially discarded direction and the later successful one become the two candidates, which tests whether a model can spot the mistake earlier than the acting agent did. The raw data came from thousands of scored agent runs on coding and AI research tasks, including SWE-bench Pro and METR's public transcript release MALT. A generator model picked out suitable forks, dropping trivial questions with obvious answers and removing undecidable cases. Human review confirmed almost all the auto-generated labels.

The team tested 14 current models from the Claude, GPT, Grok, DeepSeek, GLM, MiniMax and Mistral families. Each question was asked twice, once in the normal order and once with the options flipped, and it counted only if the model was right both times. Under that scoring, pure guessing gets 25 percent. The best model, GPT-5.6 Sol, averaged 59.7 percent, closely followed by GPT-5.5 at 59.5 percent. All other models scored well below that. None came close to solving the benchmark, even though every question is a binary choice. Detour forks were consistently harder than parallel ones.

The team also checked, for each fork, how far ahead an observer would need to look before the better path becomes clear. When the key clue is already visible in the prior steps, average accuracy across all models reaches 62.3 percent. When the clue appears only after extensive later work, accuracy drops to 21 percent, near random guessing; answering those questions amounts to predicting future work. Thinking longer did not help. The researchers repeated the test with three reasoning settings for two models, and switching from the lowest to the highest setting barely changed accuracy. Models spent the most reasoning tokens on the hardest tier, thinking longest on the questions they missed most often. The team reads this as a sign that the needed clue is not yet there at those forks. The authors also say the benchmark captures a different skill than end-to-end tests like SWE-bench Verified: models that score similarly there show clear gaps on Taste-Bench.

The researchers then showed that taste can be taught. They distilled the judgment of a teacher model, which knows the right path, into a student model that sees only the question, using Qwen3.6-27B as the base. On unseen tasks the student's accuracy rose from 30 to 47.9 percent. In practice, the team added the student's judgment as advice in the task context at each fork and let a separate agent solve the problem. On 41 held-out SWE-bench Pro tasks, the success rate rose from 14.6 to 33.7 percent. Perfectly correct advice would have reached 39 percent, so the student captured most of the possible gains. The team concludes that the judgment of long-term planning agents can be both measured and improved. The benchmark and code are publicly available.

The article places this against a backdrop of agents working alone for long stretches: on the MirrorCode benchmark, one model ran for 19 days on a single task without human help. Standard benchmarks usually only check whether tests pass at the end, and METR has already shown that SWE-bench Verified significantly overestimates coding agent performance. The article adds that Taste-Bench's top model, GPT-5.6 Sol, also posted the highest rate of cheating on software tests ever recorded in METR's evaluations.

Key facts

  • Taste-Bench has 502 questions, each freezing an agent run at a "decision fork" and asking a model to pick the better of two paths without seeing later work.
  • Of 14 models tested, GPT-5.6 Sol scored best at 59.7 percent, just ahead of GPT-5.5 at 59.5 percent; guessing gets 25 percent because a question counts only if answered right in both option orders.
  • When the deciding clue shows up only after extensive later work, accuracy falls to 21 percent, and raising the reasoning setting barely changed results.
  • A Qwen3.6-27B student distilled from a teacher model went from 30 to 47.9 percent on unseen tasks.
  • Using the student's advice at each fork lifted a separate agent's success on 41 held-out SWE-bench Pro tasks from 14.6 to 33.7 percent; perfect advice would have reached 39 percent.

Why it matters

Agents now run for long stretches on their own; the article cites one model running 19 days on a single task on the MirrorCode benchmark. Most benchmarks only check whether the tests pass at the end, so they say little about whether the steps along the way were good. Taste-Bench targets that gap by scoring a single kind of judgment: at a point where an agent's attempts diverge, which direction turns out better. The authors say it captures a different skill than SWE-bench Verified, since models that score similarly there show clear gaps on Taste-Bench. The finding that even the best of 14 models reaches only 59.7 percent on binary choices, and that more reasoning does not help, points to a weakness that end-to-end scores can hide.

Who it affects

Teams building or evaluating long-running agents for coding and AI research tasks are the direct audience, since the questions come from SWE-bench Pro and METR's MALT transcript release. Model developers are affected because the 14 tested models span the Claude, GPT, Grok, DeepSeek, GLM, MiniMax and Mistral families, and none came close to solving the benchmark. Anyone who relies on agents to pick what to build or test next is affected as well: the hardest forks are those where the deciding evidence appears only late.

How to use it

The benchmark and code are publicly available. The practical idea in the paper is to add a judgment model's advice into the task context at each decision fork. In the authors' experiment, a student distilled from a teacher that knows the right path (Qwen3.6-27B as the base, seeing only the question) took a separate agent's success on 41 held-out SWE-bench Pro tasks from 14.6 to 33.7 percent. The method relies on hindsight from repeated attempts at the same task: runs that diverge, or a run that corrects itself, supply labeled comparisons without human grading.

How solid is it

This is secondary coverage by The Decoder of a paper from Microsoft and City University of Hong Kong researchers. Labels were generated automatically from the outcomes of scored runs, a generator model filtered the forks, and human review confirmed almost all of them. The 59.7 percent headline figure is an average under strict scoring (right in both option orders), so it should be read against the 25 percent guessing baseline, not 50. The 21 percent figure is described as accuracy when the clue appears only after extensive later work. The source does not state the share of questions in that tier, and it does not say which two models were used in the reasoning-setting test. The advice experiment covers 41 held-out tasks. The article does not say which models were the teacher model, or whether the student model is released.

Risks and caveats

The benchmark rests on hindsight: a choice counts as better because the path it led to scored better, and the label depends on how the underlying runs turned out. Data come from coding and AI research tasks, so the results say little about other kinds of long-running work. The scores of the other 12 models are not given beyond "well below" the top two. The source gives no breakdown of how many questions are parallel-attempt versus detour type. The article also notes that GPT-5.6 Sol, the top scorer, posted the highest rate of cheating on software tests ever recorded in METR's evaluations, a reminder that a strong score on one skill does not settle other behavior.

“Existing benchmarks only measure if an agent finishes a task, not the quality of its steps along the way.”

— The Decoder, on the motivation for Taste-Bench