IBM's ALTK-Evolve lifts a GPT-4.1 agent to 67.5% Pass^4 on τ²-bench retail

A Hugging Face blog post from IBM Research describes ALTK-Evolve, a library that turns an agent's own training-task attempts into short behavioral rules and appends them to the system prompt. The same model, GPT-4.1, does both the guideline extraction and the agent work. No weight updates and no human annotation are involved. The authors earlier showed the approach on coding agents; this post tests it on customer service.

The test bed is τ²-bench from Sierra, where an agent talks to an LLM-simulated customer, uses domain tools (look up an order, issue a refund, rebook a flight) and must follow a written policy. A task counts as solved only if the final database state matches the expected outcome. The authors use the retail and airline domains, run the benchmark's own two-phase THINK then ACT ReAct agent with gpt-4.1 as the agent and gpt-5.5 as the customer simulator, with 4 trials per task. The only change is a markdown list of learned guidelines placed after the domain policy. Examples: authenticate the user before any order action, confirm all changes and get explicit approval before a database update, itemize refunds, take one action per step, and clean up a name before looking a user up. Guidelines are extracted from train tasks and applied unchanged to held-out test tasks.

The authors compare three setups: a baseline with no guidelines, "sup3" (only guidelines with support of at least 3, where support is the number of distinct training tasks a guideline came from), and "lossless" (every guideline kept after consolidation, support of at least 1). They track Mean Accuracy (average success across the 4 trials) and Pass^4 (share of tasks solved on all 4 trials, the metric the leaderboard reports).

With gpt-4.1 on held-out tasks, Mean Accuracy rises in retail by 4.4 points (78.1 to 82.5) and in airline by 7.5 points (57.5 to 65.0). Pass^4 rises by 10 points in both: retail 57.5 to 67.5, airline 35.0 to 45.0. The best amount of guidance differs by domain. In retail, more is better on every metric, and the small sup3 core barely beats baseline on accuracy. In airline, lossless leads on accuracy (65% vs 60%), but the compact sup3 core of 14 guidelines is more reliable, reaching 45% Pass^4 against 40% for lossless. The authors conclude that support count is a knob to tune per domain.

On cost, the authors say guidelines add input tokens, not turns. On matched runs solved under both baseline and guidelines, lossless trims about 0.3 to 0.4 assistant turns (retail 10.59 to 10.16, airline 9.15 to 8.82), and no variant lengthens conversations. The larger prompt does cost more: retail's 292-guideline prompt moves total LLM cost per conversation from about $0.084 to about $0.139, while airline's smaller playbook is cheaper. The authors expect prompt caching to claw back much of this in production, since the guideline block is identical on every call; they do not measure that here.

To check whether guidelines only patch a weak model, the authors reran airline with gpt-5.5 as both agent and extractor. No guideline reached support 3 in that setup (they top out at support 2), so the compact core is sup2. On held-out tasks, accuracy rose 10 points (72.5 to 82.5) and Pass^4 rose 15 points (55.0 to 70.0), a bigger reliability gain than gpt-4.1 got on the same tasks. They add that where the baseline's reliability swung between reruns, the lossless configuration reproduced exactly.

Leaderboard submissions are scored on the full base split (train plus test), so those figures differ from the held-out numbers above. Against an October 12, 2026 leaderboard snapshot, the post reports: retail, the gpt-4.1 agent reaches 67.5% Pass^4, ranked first, ahead of every frontier model and ahead of RAFT-30B-A3B (61.4%), a model fine-tuned on synthetic τ²-bench retail data, by 6.1 points. Airline, the gpt-4.1 agent with the sup3 core scores 44%, ranked 8th, beating GPT-5.2 (no reasoning) and nearly matching Claude Sonnet 4.5; the gpt-5.5 agent reaches 82.0%, above every frontier entry (66 to 72%). The authors close by arguing that prompt-level guidelines mined automatically from an agent's own runs, with no human annotation and no test-set leakage, can match or beat a fine-tuned system without touching weights.

Key facts

A stock τ²-bench ReAct agent running gpt-4.1, with learned guidelines appended to its system prompt, reaches 67.5% Pass^4 on retail, up from 57.5% on held-out tasks, and the authors rank it first on the base split.
On the retail base split it beats RAFT-30B-A3B, a model fine-tuned on synthetic τ²-bench data (61.4% Pass^4), by 6.1 points.
In airline, the gpt-4.1 agent with a 14-guideline core scores 44% Pass^4 (8th); with gpt-5.5 the agent reaches 82.0%, above every frontier entry (66 to 72%).
Guidelines do not lengthen conversations: lossless trims about 0.3 to 0.4 turns, but retail's 292-guideline prompt raises cost per conversation from about $0.084 to about $0.139.
The best number of guidelines depends on domain: retail gains most from the full set, airline reliability is better with the small sup3 core.

Why it matters

The post argues that an agent can get meaningfully better without fine-tuning, using only rules mined from its own earlier attempts and placed in the prompt. The authors set this against fine-tuning, which they call expensive, brittle and poor at transferring to the next domain. The headline result is that an older non-reasoning model with a learned playbook outscored a fine-tuned system on retail and every frontier entry on the October 12, 2026 snapshot. The gpt-5.5 airline run suggests the gain is not limited to weaker models.

Who it affects

Teams building customer-service or other policy-bound tool-using agents, especially those who cannot or do not want to fine-tune. It also concerns anyone who reads τ²-bench leaderboards, since the post shows a prompt-only entry competing with frontier models and a fine-tuned one.

How to use it

The ALTK-Evolve library is linked at github.com/AgentToolkit/altk-evolve, and the benchmark at github.com/sierra-research/tau2-bench. The recipe in the post: run the agent on training tasks, have the model distill successes and failures into short rules, then append them as a list after the domain policy. Tune how many guidelines to keep by support count for each domain rather than hard-coding it. Expect a larger prompt and higher input cost; the authors expect prompt caching to recover much of that. No release plan beyond the GitHub link is given.

How solid is it

The numbers are the authors' own, run on the public benchmark, and the post links leaderboard pull requests (PR #542 for retail, PR #551 for airline); the text does not confirm they were accepted. Comparisons are against a single October 12, 2026 snapshot. Held-out and base-split figures are different measurements (for example, gpt-5.5 airline is 70.0% Pass^4 on held-out tasks and 82.0% on the base split, which includes training tasks). Several tables and figures are not reproduced in the text. The post states no confidence intervals, significance tests or test-set size.

Risks and caveats

Guidelines raise prompt size and cost, and the cost drop from caching is an expectation, not a measurement. More guidance is not always better: in airline the full set was less reliable than the 14-guideline core, so settings need per-domain tuning. The base-split leaderboard scores include the training tasks the guidelines were mined from, which is why they differ from the held-out numbers. The result is on two customer-service domains with a gpt-5.5 simulated customer, and the post does not show it holds elsewhere.

“support count is a knob worth tuning per domain, not a value to hard-code”

— ALTK-Evolve post, Hugging Face blog