Glonce

IBM Research's ALTK-Evolve matches ACE's agent accuracy at a fraction of the cost

agentsThe Glonce editorsHugging Face🔥72Glonce rating72Impact65Novelty65Relevance95Surprise55Players55Credibility85

A blog post published on Hugging Face under IBM Research's account introduces ALTK-Evolve, a system that lets an LLM agent learn from its own past task attempts and compares it head to head against ACE (Agentic Context Engineering), an existing agentic-memory system built on the same idea. Both systems turn an agent's past trajectories into reusable lessons and feed them back at inference time, with no weight updates and no human labels. The premise is that when an agent fails a realistic multi-step task, such as splitting a bill, finding a song, or reconciling an order across nine simulated apps, the failure is usually not a knowledge gap: the model mis-paginates an API, resolves the wrong person, or returns a value nobody asked for. That kind of procedural know-how is learnable from the agent's own history.

The two systems agree on the hard part: neither compresses the lessons it collects into a short summary. ACE calls the failure modes of compression brevity bias, where optimization collapses toward short generic instructions, and context collapse, where a model rewriting its whole context each step summarizes away the useful detail; its fix is a rich, itemized playbook with a helpful or harmful counter on every bullet. ALTK-Evolve reaches the same conclusion from its own direction, keeping a support count on every distinct guideline, tracking how many independent episodes produced it, rather than summarizing the store down to a handful of rules.

Where the two systems differ is consolidation and delivery. ACE grows one playbook through a Generator, Reflector, Curator loop with incremental delta updates and de-duplication by embedding. ALTK-Evolve clusters near-duplicate lessons and merges within a cluster in a support-conserving way, so a merged guideline keeps the combined count of the lessons behind it, and it extracts typed guidelines (strategy, recovery, and optimization) with provenance back to the source trajectory. The bigger difference is delivery: ACE injects its comprehensive playbook on every step, the same way regardless of model or task. ALTK-Evolve treats delivery as a dial rather than a constant, sending a small fixed core of high-support guidelines plus a handful selected per task, or the full consolidated set when a model has the headroom to use it.

On the AppWorld benchmark (the test_normal split, 168 tasks, using the same ReAct code agent for both systems), the delivery difference shows up directly in inference cost. On the strong base model, DeepSeek-V3.2, ALTK-Evolve is better than ACE on both benchmark metrics while spending about 40% of ACE's inference cost. On the weaker base model, gpt-oss-120b, ALTK-Evolve scores 56.0 versus ACE's 54.8 on Task Goal Completion (TGC), a gap the authors call a tie given the benchmark's run-to-run noise; a repeat ALTK-Evolve run landed at 54.8, matching ACE almost exactly, and the accuracy tie held at about one-seventh of ACE's cost.

A by-difficulty breakdown shows the two models behave differently. On gpt-oss-120b, ACE's full playbook has the edge on Easy and Medium tasks, where generic instruction-following covers most of the task, but ALTK-Evolve's per-task retrieval wins the Hard tier and, with it, the overall aggregate. On DeepSeek-V3.2 the pattern flips: ACE only edges ALTK-Evolve on Medium tasks, while ALTK-Evolve leads Easy, Hard, and Overall, since the stronger model has more capacity for more lessons to keep helping instead of crowding each other out. Each model is given its own best configuration: the full consolidated guideline set for the strong model, selective retrieval for the weaker one, because a large context overwhelms a weaker model rather than helping it.

On methodology, the post notes that ACE's own published results use a different base model (DeepSeek-V3.1), so IBM Research ran ACE itself in-house on the same AppWorld splits and the same base models as ALTK-Evolve to keep the comparison controlled. Both systems use the same ReAct agent and differ only in the prompt template, which is why the two no-memory baselines the post reports overall differ (72.0 versus 79.8 TGC); the post does not say in the text which baseline belongs to which system, and it says the comparison should rest on same-or-better accuracy at a fraction of the tokens rather than on that baseline gap. Results are single pass@1 runs, standard for this benchmark. The post closes by pointing readers to the ALTK-Evolve library, which includes the extraction, consolidation, and retrieval pipeline used in the comparison, and to a full technical report for the complete method and ablations.

Key facts

  • ALTK-Evolve, from IBM Research, and ACE (Agentic Context Engineering) are both agentic-memory systems that turn an LLM agent's past task trajectories into reusable lessons without weight updates or human labels.
  • On the strong base model, DeepSeek-V3.2, ALTK-Evolve beat ACE on both AppWorld metrics while using about 40% of ACE's inference cost.
  • On the weak base model, gpt-oss-120b, ALTK-Evolve scored 56.0 versus ACE's 54.8 on Task Goal Completion, a gap the authors call a tie within run-to-run noise, at about one-seventh of ACE's cost.
  • Both systems refuse to compress lessons into a short summary and instead track a support count per guideline; they differ in delivery, ACE injects its whole playbook every step while ALTK-Evolve retrieves a handful of guidelines per task or the full set when a model can use it.
  • The comparison used the AppWorld test_normal split (168 tasks) with the same ReAct code agent for both systems; their reported no-memory baselines differ (72.0 vs 79.8 TGC) because of a prompt-template difference, not because of memory.

Why it matters

Agentic memory is a way to make LLM agents better at multi-step tasks without retraining the underlying model: instead of fine-tuning, the agent accumulates lessons from its own past attempts and gets them fed back at inference time. ALTK-Evolve and ACE agree on the harder design question, that these lessons should not be compressed into a tidy summary, since compression is exactly what erases the specific, hard-won detail that makes a lesson useful. What ALTK-Evolve adds is the observation that even once you accept that principle, how much of that uncompressed memory you actually send to the model on a given step is a separate knob, and that knob is what determines the token bill. Treating delivery as a dial rather than a fixed injection is the concrete change behind the reported cost savings.

Who it affects

Teams building or operating LLM agents that need to improve from experience without retraining a model, and who are sensitive to per-token inference costs, since every step of a memory-augmented agent pays for whatever context is injected. It is particularly relevant for teams running agents on weaker or cheaper base models, where the post argues a large injected context can overwhelm the model rather than help it, versus stronger models that have the headroom to benefit from more of the guideline set.

How to use it

The post points readers to the ALTK-Evolve library, which includes the extraction, consolidation, and retrieval pipeline used in the reported comparison, and to a full technical report covering the complete method and ablations. No pricing, licensing terms, or release stage are given in the source beyond pointing to the library and the report.

How solid is it

The comparison is IBM Research's own in-house evaluation, not an independent benchmark: the post explains that it re-ran ACE itself on the same AppWorld splits and the same base models as ALTK-Evolve, rather than citing ACE's own published numbers, because ACE's original paper used a different base model. Results are single pass@1 runs, standard for AppWorld but without confidence intervals. On the weaker model, the authors themselves describe the 56.0-to-54.8 gap as a tie within the benchmark's run-to-run noise rather than a clear win, and a repeat run of ALTK-Evolve landed at 54.8, exactly matching ACE. The two no-memory baselines used for context (72.0 and 79.8 TGC) are not individually labeled by system in the text, which limits how precisely readers can attribute the baseline gap.

Risks and caveats

All results come from the system's own developers comparing their own reimplementation of a competing system, on one benchmark (AppWorld) and two base models, with single-run pass@1 evaluation rather than repeated trials with variance reported. The accuracy advantage on the weaker model is explicitly a tie, not a win; the token-cost savings are the more solid claim there. The two systems' no-memory baselines differ because of a prompt-template difference between them, a variable the post flags but that readers cannot independently disentangle from the reported memory gains. The article text itself does not name an individual author or a publication date; the IBM Research attribution here follows from the Hugging Face account the post is published under, not from a byline in the text.

“We treat delivery as a dial, not a constant.”

— ALTK-Evolve blog post, IBM Research