IBM Research's ALTK-Evolve calibrates agent memory by model

IBM Research evaluated ALTK-Evolve, a system that lets an AI agent learn from its own past task attempts without any weight updates or human annotation, across eight language models ranging from a 30B dense model to frontier proprietary systems. The method works by having the agent attempt tasks, extracting behavioral guidelines from both its successful and unsuccessful runs, consolidating those into a reusable set, and then injecting either the full set or a task-relevant selection of it back into the agent's context at inference time.

Testing ran on AppWorld, a benchmark of 585 multi-step tasks (168 in test_normal, 417 in test_challenge) spread across 9 simulated apps such as calendars, messaging and payments. Both memory configurations drew from a guideline set mined once from AppWorld's training split only, so no test-split data went into building it. Results were scored two ways: Task Goal Completion (TGC), the share of individual tasks fully and correctly completed, and Scenario Goal Completion (SGC), a stricter all-or-nothing measure that only counts a scenario as passing if the agent succeeds on every variant of it.

The evaluation surfaced three recurring patterns tied to model capability rather than parameter count alone. Strong models with headroom benefited most from the full guideline set: DeepSeek-V3.2 (671B MoE) climbed +9.5 percentage points in TGC and +16.1 percentage points in SGC when given every guideline it had mined, including rare edge cases. Weaker or smaller models were drowned by that same volume and did better with a compact, high-confidence core plus a handful of guidelines retrieved per task: gpt-oss-120b (117B MoE) gained +16.1 percentage points in TGC this way, at only about 5% more tokens than the no-memory baseline, while giving it the full guideline set instead produced a smaller gain and cost roughly 50% more tokens. A third group, exemplified by GLM-5 (745B MoE), showed no measurable gain from either configuration; the researchers call this the 'saturated' pattern but are explicit that the label describes what they observed, not a proven cause, since headroom, context-window size, architecture, guideline quality and task distribution could all be contributing factors that remain unseparated.

Memory kept paying off even near the top of the range: GPT-5.5 and Opus, both close to the TGC ceiling, still gained +7.2 and +7.1 percentage points respectively on the stricter SGC metric, because good guidelines particularly help an agent clear every variant of a scenario rather than just the average case. On cost, DeepSeek ran roughly the same number of ReAct steps with memory as without (about 18 to 19 on average), meaning the extra expense came from inflated per-step input tokens rather than longer trajectories; the researchers note that prompt caching can substantially cut that cost in production, since the shared guideline-set prefix is identical across steps.

IBM Research frames the next steps as: building a learned selector to replace the current cosine-similarity retrieval, which it has already shown does not perfectly predict which guidelines help a given task; addressing very weak models, where self-distillation lacks signal below some minimum capability baseline; validating the approach on benchmarks beyond AppWorld; and running controlled experiments to isolate the role of context-window size from raw model capability. The ALTK-Evolve library and a fuller technical report are both referenced as available for readers who want the complete method and ablations.

Key facts

DeepSeek-V3.2 (671B MoE) gained +9.5 percentage points in task completion (TGC) and +16.1 percentage points in scenario completion (SGC) when given its full self-mined guideline set.
gpt-oss-120b (117B MoE) gained +16.1 percentage points in TGC using curated, per-task retrieval at only about 5% more tokens than baseline, versus roughly 50% more tokens and a smaller gain for the full guideline set.
GLM-5 (745B MoE) showed no measurable gain from either memory configuration, the pattern the researchers label 'saturated'.
The evaluation ran on AppWorld: 585 multi-step tasks (168 test_normal, 417 test_challenge) across 9 simulated apps, scored by Task Goal Completion and the stricter Scenario Goal Completion.
GPT-5.5 and Opus, both near the ceiling on TGC, still gained +7.2 and +7.1 percentage points respectively on SGC, showing memory keeps helping even strong models.

Why it matters

The result cuts against the simple assumption that more distilled experience is always better for an agent. IBM Research's own framing is that agentic memory is a dose to calibrate, not a feature to switch on: dumping a full guideline set into every model's context can waste tokens on weaker models without improving their results, while starving strong models of edge-case guidance leaves performance on the table. Getting the dose right, tier by tier, is what turns a memory system from a blanket add-on into something that actually moves task completion.

Who it affects

Anyone building agents that are meant to improve from their own trial and error, particularly teams working with a mix of model sizes or open-weight models in the gpt-oss-120b class rather than only frontier proprietary systems. Because the method needs no weight updates and no human-labeled data, it is aimed at practitioners who want a cheap, portable way to add self-improvement to an existing agent stack rather than researchers doing model training.

How to use it

The pattern to apply is: strong, high-headroom models get the full mined guideline set; weaker or smaller models get a compact core plus a handful of guidelines retrieved per task, which was both the most accurate and the cheapest option in the gpt-oss-120b case; and already-saturated models get no extra context spent on memory until their remaining failure modes are better understood. In production, keeping the shared guideline-set prefix stable so it stays cacheable is the lever that keeps even the full-set approach affordable. IBM Research points readers to the ALTK-Evolve library, which includes the extraction, consolidation and retrieval pipeline used in the evaluation, plus a fuller technical report for the complete method and ablations.

How solid is it

The findings rest on a single benchmark, AppWorld, run across eight models spanning a 30B dense model to frontier proprietary systems, with guidelines mined only from the training split and evaluated on held-out test tasks. The percentage-point gains are the study's own reported numbers rather than external replication, and the authors are careful to flag that the 'saturated' pattern is a description of what they observed in GLM-5's case, not a proven explanation of why it happened; they list several unseparated candidate factors, including context-window size and guideline quality, as open questions. The team says broader benchmarks and real-world deployments are still in progress.

Risks and caveats

The write-up carries no named human author and no publication date, and several natural follow-up numbers are explicitly not given: there is no numeric TGC or SGC value for GLM-5's non-gain, no quantified figure for how much smaller gpt-oss-120b's full-guideline-set gain was, and no stated numeric threshold for the 'minimum capability baseline' below which the authors say self-distillation lacks signal for very weak models. The retrieval method itself is also flagged as imperfect: the authors state that ranking guidelines by cosine similarity does not reliably predict which guidelines actually help on a given task, which is why they list a learned selector as future work rather than a solved problem.

“Agentic memory is not a feature you switch on. It's a dose you calibrate to the model.”

— IBM Research, ALTK-Evolve blog post