Memory substrates for LLM agents: no single winner, study finds

Memory is becoming core infrastructure for long-horizon LLM agents, systems that keep working and remembering across many steps rather than a single exchange. The paper argues that existing evaluations give limited guidance on which memory substrate, the underlying medium in which memory is represented and stored, should be used under different operating conditions. To address that gap, the authors built a controlled harness that evaluates a broad range of substrate types side by side: dense and sparse retrieval indices, plain text records, structural stores, hierarchical stores, refinement-based memories that revise what they hold over time, parametric updates that write memory directly into the model's weights, and mechanisms compatible with the model's own activation or context state.

The comparison runs across three backbone models and four benchmark suites, covering both user-centric question answering, where an agent must recall facts from a long history, and agent-centric decision-making, where it must choose and take the right action in sequence. Under this single unified harness, the authors instrument 26 separate performance and efficiency metrics per configuration. The abstract does not name which three models or which four benchmarks were used, and it does not publish the individual metric values, so the size of the effects described below is not visible from the abstract alone.

The central finding is that no single substrate consistently dominates. Broad retrieval, pulling in more of the stored memory when answering, benefits long-context factual question answering. But that same breadth can actively harm sequential decision-making, because it shifts the model's attention away from the context that is actually critical to the next action it needs to take. Scalability adds a further complication: a substrate that performs well when the interaction history is of moderate length can become costly to run, or simply unreliable, once that history grows much longer.

Because no fixed substrate wins across every regime, and the right choice shifts with how long the history gets, the authors conclude that substrate routing, switching which memory backend an agent relies on depending on the situation, is a necessary component of adaptive agent memory systems, not an optional extra. They frame the results as empirical guidance for designing memory that is efficient, reliable and matched to the operating regime it will actually run in. The code behind the harness is not public yet: the authors say it will be released only once the paper is accepted.

Key facts

The evaluation runs across three backbone models and four benchmark suites, covering both user-centric question answering and agent-centric decision-making, while tracking 26 performance and efficiency metrics per configuration.
No single memory substrate consistently wins: broad retrieval helps long-context factual question answering but can hurt sequential decision-making by pulling attention away from action-critical context.
Scalability is a separate failure mode: substrates that work well at moderate history lengths can become costly or unreliable once the history grows much longer.
The authors conclude that adaptive substrate routing, switching which memory backend is used depending on the situation, is a necessary component of agent memory systems rather than an optional extra.
The harness's code is not public yet; the authors say it will be released only once the paper is accepted.

Why it matters

Memory is becoming core infrastructure for long-horizon LLM agents: systems that act across many steps rather than answering a single prompt. Many teams building such agents default to one memory scheme, often a retrieval index, and assume it will generalize. This evaluation is a controlled, apples-to-apples comparison across a broad range of substrate types in one harness, and its headline result works against that assumption: no single substrate wins across the tested conditions, and a choice that helps one kind of task, recalling facts, can actively hurt another, taking the right sequential action.

Who it affects

Anyone designing the memory layer for a long-running LLM agent, coding assistants, customer-facing assistants, or other systems that need to recall and act on a long interaction history, is affected by this result. It also matters for teams building agent frameworks or memory-as-a-service offerings around a single default substrate, since the paper's finding suggests that default will underperform in some regime it was not tuned for.

How to use it

The practical takeaway is not to assume that wider retrieval is always safer: it helps long-context factual recall but can actively hurt tasks where the agent has to choose and sequence actions, by diluting its attention with retrieved content that is not action-critical. For agents that need to operate over very long histories, the evidence here argues for testing substrate cost and reliability explicitly at the history lengths the system will actually see, rather than assuming that what worked at moderate length keeps working. The authors' own suggested fix is to route between substrates adaptively rather than commit to one for every regime. There is nothing to install yet: the code is withheld until the paper is accepted.

How solid is it

The design is reasonably broad for this kind of comparison: three backbone models, four benchmark suites and 26 tracked metrics under one unified harness. But the abstract published on Hugging Face Papers does not name which three models or which four benchmarks were used, and it gives no actual metric values, so the size of the effects it describes cannot be checked from what is public here. No peer-review or acceptance status is stated beyond the fact that code release is conditioned on acceptance, and the code itself is not out yet, so independent reproduction is not currently possible.

Risks and caveats

Because no substrate-by-substrate numbers are given in the abstract, readers cannot yet judge how large the gaps between substrates actually are in a specific setting, or how often no single substrate wins outright versus one being close enough by default. The paper's own proposed fix, adaptive substrate routing, is stated as a conclusion rather than demonstrated with a working routing system in the abstract, so it should be read as a research direction rather than an available technique.

“Our results show that no single substrate consistently dominates”

— the paper's authors