Glonce

Taste-Bench finds frontier LLM agents get only 59.7% of long-horizon calls right

researchThe Glonce editorsHugging Face🔥61Glonce rating61Impact68Novelty72Relevance95Surprise65Players40Credibility82
Taste-Bench finds frontier LLM agents get only 59.7% of long-horizon calls right

Researchers introduce the idea of an LLM agent's "taste": its ability to make good decisions over the course of a long-horizon task, such as which hypothesis to test or which implementation to build on, where those choices shape the outcome of the entire run. They argue that while existing benchmarks measure whether an agent succeeds at a task end to end, none of them isolates and measures this decision-making ability on its own. To fill that gap, they build Taste-Bench, a benchmark of "taste questions" constructed automatically from trajectories that agents produced while working on engineering and research tasks. Each question presents a decision fork: a point in a trajectory where several directions are available and only one of them leads to a better outcome. The model being evaluated has to pick among these directions without seeing what happens afterward. The forks are mined automatically, either from parallel attempts at the same task or from detours inside a single trajectory, so the benchmark requires no human annotation.

When the authors evaluate frontier models on Taste-Bench, the best-performing model answers only 59.7% of the questions correctly. They also find that forks are much harder for every model when the evidence that would decide the right answer appears later in the trajectory, and that giving a model a larger reasoning budget does not improve its accuracy on the benchmark. Finally, they show that taste can be trained: distilling the judgment of a teacher model that has already seen the outcome into a student model makes the student's decisions better on unseen tasks and improves its end-to-end success rate on held-out SWE-bench Pro tasks, though no numeric figure for that improvement is given.

Key facts

  • Taste-Bench is a new benchmark that measures an LLM agent's "taste", its ability to make good decisions on long-horizon engineering and research tasks.
  • Questions are built automatically from decision forks mined from agent trajectories, without human annotation, drawing on parallel attempts at a task and detours within a single trajectory.
  • The best-performing frontier model evaluated answers only 59.7% of Taste-Bench questions correctly.
  • Forks are much harder when the deciding evidence appears later in the trajectory, and a larger reasoning budget does not raise accuracy.
  • Distilling judgment from a teacher model that saw the outcome into a student model improves the student's decisions on unseen tasks and its end-to-end success on held-out SWE-bench Pro tasks.

Why it matters

As LLM agents take on longer engineering and research tasks, the intermediate choices they make along the way, not just whether they finish the task, increasingly determine the outcome. Existing benchmarks only score end-to-end success, so they cannot tell whether an agent failed because it lacked skill or because it made a poor call at some fork in the road. Taste-Bench is built specifically to isolate and measure that decision-making ability.

Who it affects

The work is aimed at teams building and evaluating agentic systems for coding and research, who need a way to check an agent's judgment separately from whether it happened to land on the right final answer. It also matters to anyone training agents, since the paper suggests this judgment can be taught rather than only observed.

How to use it

Taste-Bench works as an evaluation tool: questions are generated automatically from an agent's own trajectories, using parallel attempts at the same task or detours within one run, so no manual labeling is needed to build a test set for a given agent or task type. The paper's distillation approach, feeding a student model the judgment of a teacher that already saw how a trajectory turned out, is presented as a method for improving an agent's decisions on tasks it has not seen before, including measurable gains on held-out SWE-bench Pro tasks.

How solid is it

The account here is drawn directly from the paper's own description of its method and findings. It does not name the specific frontier models tested, the authors or their affiliations, a publication date, or the number of questions or trajectories in Taste-Bench, so those details cannot be confirmed from what is available.

Risks and caveats

The paper does not report a numeric figure for how much the distilled student model improves on SWE-bench Pro, only that it improves. It also does not identify which frontier models were evaluated, which limits how directly the 59.7% figure can be compared against known model rankings on other benchmarks.

“We refer to the ability to make good long-horizon decisions as the taste of an agent.”

— the authors, Taste-Bench paper