Glonce

ALTK-Evolve cuts a GPT-4.1 agent's consistency gap roughly in half

agentsThe Glonce editorsHugging Face🔥77Glonce rating77Impact65Novelty72Relevance100Surprise68Players68Credibility88
ALTK-Evolve cuts a GPT-4.1 agent's consistency gap roughly in half

A Hugging Face blog post lays out a specific, measurable failure mode in AI agents: the same request, run again, does not always get the same result. Standard benchmarks report Mean@k, the average pass rate across k repeated runs of a task, and that is the number nearly every agent leaderboard shows. It does not answer what a real user actually needs to know: will the agent still solve my exact request if I ask it twice? That second question is Pass^k, the share of tasks the agent solves on every one of k runs, and it is always less than or equal to Mean@k. On AppWorld, a benchmark of simulated multi-step tasks across apps such as calendars, messaging and payments, a ReAct agent built on GPT-4.1 posts a Mean@5 of 77.4% across five repeated runs, genuinely strong, but its Pass^5 is only 53.0%: nearly a quarter of the benchmark consists of tasks the agent can sometimes solve and sometimes cannot, with nothing about the task itself changing between runs. The post calls the difference, Mean@k minus Pass^k, the consistency gap. In aggregate it is 24.4 percentage points, and on the benchmark's hardest tier it reaches 30 percentage points.

The explanation offered is about the shape of the model's token-probability distribution at each decision point in a trajectory, such as which API to call, what argument to pass, or whether to retry. A sharp distribution puts most of its probability mass on one token and reliably produces the same choice; a flat one splits mass across several near-tied options, so which one wins is close to a coin flip. Small platform-side noise, such as GPU floating-point non-associativity or request batching, is not enough to reorder a sharp winner but is enough to flip a near-tie, and because a trajectory chains dozens of such decisions, a small per-step chance of flipping compounds into a large chance that some run goes differently. Fixed seeds and greedy decoding do not fix this: they govern how a distribution becomes a token, not the shape of the distribution itself, so probabilities on a hosted endpoint can still drift slightly run to run even at temperature zero. The evaluation itself ran the ReAct agent at temperature 0.0, so this variance is not from ordinary sampling.

The fix is a two-stage pipeline added to the existing ALTK-Evolve system, which had previously turned an agent's own past trajectories into reusable guidelines fed back into its context at inference time, without retraining any model weights. The new piece is a diagnostic called the Consistency Analyzer: given a single recorded trajectory, it replays each decision step through controlled resampling, one extra model call per step requesting several completions at once (5 by default), scored against the context already recorded rather than through new tool calls, new environment interactions, or a second full run of the task. That makes detection fully black-box: no logits, no model internals, no instrumentation beyond the trace already on hand, and it produces a consistency score per step that flags exactly which decisions are at risk of flipping on the next run. Every flagged step becomes a candidate consistency guideline in ALTK-Evolve's standard format. One worked example, mined from an AppWorld task asking how many activities are checked off in a SimpleNote bucket list, produced two guidelines: use a line-anchored regex rather than a plain substring count when tallying checkbox markers, because note titles often repeat the marker symbol in a legend line, and always verify search results for note queries by checking for multiple matches before proceeding. A linked two-minute demo shows five parallel runs of that one task splitting 3-2 because of the agent's uncertainty about the counting strategy, then agreeing across all five runs once the guidelines are added to its context.

Across the full AppWorld test_normal split, 168 tasks, adding consistency guidelines cut the aggregate gap roughly in half. Pass^5 rose from 53.0% to 69.0% while Mean@5 still rose, from 77.4% to 81.0%, so the gain in reliability did not come at the cost of average accuracy, a requirement the post treats as non-negotiable: trading Mean@5 away for a higher Pass^5 would only shift unreliability around rather than remove it. The consistency gap itself narrowed from 24.4 to 12.0 percentage points, and nearly a third of the tasks that had been inconsistent at baseline became tasks the agent now passes on every run. Gains were uneven by difficulty: the medium tier improved most, +22.9 percentage points, a 44% relative gain, the hard tier gained +14.3 percentage points, 45% relative, and the easy tier, which had the least room to improve, gained +12.2 percentage points. The guidelines were not just memorizing the one trajectory they were mined from: applied to a different, related task in the same AppWorld scenario, they still lifted Pass^5 by 13.0 percentage points, only 3 points below the 16.0-point gain on the same task. A second, weaker model, gpt-oss-120b, showed the same pattern from a much lower starting point: its same-task Pass^5 rose from 10.1% to 16.1%, up 6.0 percentage points, and its similar-task generalization gain, 8.7 percentage points, actually exceeded its own same-task number.

The post frames this as a warning against treating agent evaluation as solved once Mean@k looks good: a stronger model raises Mean@k, it argues, but does not necessarily shrink the consistency gap, because inconsistency is an axis orthogonal to raw capability. Its practical advice for anyone shipping an agent: report Pass^k alongside Mean@k, expect the gap to widen on harder tasks, and reach for diagnosis rather than a bigger model first. The Consistency Analyzer needs no grader and no live replay, just one extra LLM call per decision step, which the post argues makes it usable on production traffic, where a task often cannot be replayed end to end even once. All of the quantitative results come from a single benchmark split, AppWorld's test_normal; the post reports no numbers from other benchmarks or from a live production deployment, and it does not state the cost, latency or compute overhead of running the Consistency Analyzer itself beyond the qualitative description of one additional model call per decision step, nor how many decision points a typical trajectory contains. The ALTK-Evolve toolkit, now including the Consistency Analyzer and consistency-guideline generation, is open source on GitHub, and a technical report with the full methodology and evaluations is posted to arXiv.

Key facts

  • A GPT-4.1 ReAct agent on the AppWorld benchmark posts a strong 77.4% Mean@5 (average pass rate over 5 runs) but only 53.0% Pass^5 (share of tasks it solves on every one of 5 runs), a 24.4 percentage-point consistency gap that widens to 30 points on the hardest tasks.
  • The new Consistency Analyzer diagnoses this from a single recorded trajectory: one extra model call per decision step (5 completions by default), no ground truth and no re-running the task, that flags exactly which steps are prone to flipping on a repeat run.
  • Turning flagged steps into consistency guidelines, fed back into ALTK-Evolve's existing context pipeline, narrows the aggregate gap on AppWorld's 168-task test_normal split from 24.4 to 12.0 percentage points: Pass^5 rises from 53.0% to 69.0% while Mean@5 still rises, from 77.4% to 81.0%.
  • Gains concentrate on harder tasks: the medium tier improves +22.9 percentage points and the hard tier +14.3, versus +12.2 for easy tasks that had less room to gain.
  • The guidelines generalize beyond the trajectory they were mined from, lifting Pass^5 by 13.0 percentage points on a related but different task (versus 16.0 on the same task), and reproduce on a second, weaker model, gpt-oss-120b, whose same-task Pass^5 rises from 10.1% to 16.1%.

Why it matters

Most agent benchmarks report only Mean@k, the average pass rate across repeated runs, the number that shows up on almost every leaderboard. That number cannot answer the question a real user actually has: will the same request succeed again if I ask it twice? On AppWorld, a GPT-4.1 ReAct agent looks strong by that average, 77.4% Mean@5, yet only 53.0% of tasks pass on all 5 runs (Pass^5), a 24.4 percentage-point consistency gap that reaches 30 points on hard tasks. The post treats this as a reliability problem distinct from raw capability: a stronger model raises Mean@k but does not necessarily shrink the gap, because inconsistency comes from near-tied decisions inside an agent's own reasoning, not from a lack of skill. For work where a wrong answer on even one of several identical runs is unacceptable, such as reconciling a financial transaction or checking a contract for an obligation, this gap is precisely what an averaged accuracy number hides.

Who it affects

Teams building or evaluating AI agents that field the same kind of request more than once, and who currently judge them by a single averaged accuracy number. The method here is demonstrated specifically on ReAct-style agents, GPT-4.1 and, in a second test, the weaker gpt-oss-120b, performing multi-step tasks on the AppWorld benchmark, but the diagnostic itself works on any already-recorded agent trajectory: it is black-box, needs no logits or model internals, and needs no ground-truth answer, which the post argues makes it usable directly on production traffic rather than only in a controlled benchmark, where replaying a task end to end even once is often not possible. Anyone currently relying on a single Mean@k leaderboard number to pick or ship an agent is the most direct audience, since the post's central claim is that such a number can hide a large share of tasks the agent is not actually reliable on.

How to use it

The Consistency Analyzer takes one already-recorded agent trajectory and replays each decision step through controlled resampling: one additional model call per step, requesting several completions at once, 5 by default, scored against the context already recorded rather than through new tool calls, new environment interactions, or a second full run of the task. That yields a consistency score per step, collected into a scorecard that flags exactly which decisions are prone to flipping on a repeat run. Each flagged step becomes a candidate consistency guideline written in ALTK-Evolve's existing format, so it slots into the same storage and retrieval pipeline that already feeds guidelines back into an agent's context at inference time, with no retraining of model weights. One worked example, mined from an AppWorld task about counting checked items in a SimpleNote list, produced two guidelines: use a line-anchored regex rather than a plain substring count for checkbox markers, since note titles often repeat the marker symbol in a legend line, and always verify search results for note queries by checking for multiple matches before proceeding. The Consistency Analyzer and the consistency-guideline generation it drives are now part of the open-source ALTK-Evolve repository on GitHub, alongside a technical report on arXiv with the full methodology.

How solid is it

The headline numbers come from one evaluation: a GPT-4.1 ReAct agent on AppWorld's test_normal split, 168 tasks, each run 5 times before and after adding consistency guidelines generated from a single baseline trajectory per task. Pass^5 rose from 53.0% to 69.0% and the gap between Mean@5 and Pass^5 fell from 24.4 to 12.0 percentage points, while Mean@5 itself still rose, from 77.4% to 81.0%, rather than trading accuracy for consistency. Gains were checked for generalization, not just memorization: guidelines mined from one task still lifted Pass^5 by 13.0 percentage points when applied to a different, related task in the same scenario, only 3 points below the 16.0-point gain on the task they came from, and a second run on a weaker model, gpt-oss-120b, showed the same pattern from a much lower baseline, 10.1% to 16.1% same-task, with an even larger 8.7-point gain on the generalization test. All of this is one benchmark family; the post reports no results on any other benchmark or in live production.

Risks and caveats

Every number here comes from a single benchmark, AppWorld's test_normal split; there is no result yet from another benchmark or from a live production system. The post does not state the cost, latency or compute overhead of running the Consistency Analyzer itself, only that it costs one additional model call per decision step, and it does not say how many decision points a typical trajectory contains, so the total extra-call cost per trace is not given. Two figures are easy to conflate with the aggregate 168-task results but describe something narrower. The claim that nearly a third of previously inconsistent tasks now pass every run is a share of only the subset that was already inconsistent at baseline, not of all 168 tasks. And the demo video, where five runs split 3-2 and then all agreed, is a single example task, not an aggregate result. No relative-percent figure is given for the easy tier's +12.2 percentage-point gain, unlike the medium and hard tiers. The diagnostic also still starts from one recorded trajectory per task, generated by GPT-4.1 in the published example; the post does not say how guideline quality holds up on trajectories from domains further from AppWorld's simulated apps.

“This isn't a capability problem you fix with a bigger model. It's an orthogonal axis: an agent can be capable and inconsistent at the same time.”

— the post