Glonce

Memory Tax paper finds many agent memory systems lose to naive RAG

researchThe Glonce editorsHugging Face🔥65Glonce rating65Impact55Novelty60Relevance100Surprise50Players10Credibility75
Memory Tax paper finds many agent memory systems lose to naive RAG

A paper on the "memory tax" of agent memory asks a basic question about long-horizon agents: is building a memory layer actually better than retrieving directly from raw history? The authors say the net value of memory over direct retrieval "remains largely unclear".

To test it, they ran a large-scale evaluation across multiple benchmarks and model backbones. Their finding is that many dedicated memory systems underperform naive RAG in many cases, which they read as a negative net value. The abstract does not name the systems, benchmarks or backbones, and it gives no numeric results.

Using RAG as the reference point, the authors split the net value of memory into two parts: gains over direct retrieval, which they call memory benefit, and losses, which they call memory tax. Both arise because a memory process changes how raw information is organized and accessed. The authors describe these changes as double-edged deviations from the raw history. Such deviations are designed to help the task, for example by reorganizing facts or distilling preferences, but some may also remove or obscure evidence. That is what gives rise to memory tax.

Because memory benefit reflects the expected gains of a system's design, the authors focus more on the overlooked memory tax. They study how it arises from the system side and from the evaluation side, and they generalize the concept to more application scenarios of memory.

The authors hope the notion will give a new lens and move attention from standalone performance to the net value of memory, which they say should inform more reliable memory designs and evaluations. They say they will open-source the code and results; a project page is listed at hexuan21.github.io/MemoryTax.

Key facts

  • Large-scale evaluation across multiple benchmarks and model backbones found many dedicated memory systems underperform naive RAG in many cases.
  • The authors read this as a negative net value of memory over direct retrieval from raw history.
  • They split memory's net value into memory benefit (gains over direct retrieval) and memory tax (losses), with RAG as the reference.
  • Memory tax arises when some deviations from raw history, such as reorganizing facts or distilling preferences, remove or obscure evidence.
  • The authors say they will open-source code and results; a project page is linked.

Why it matters

Memory layers are increasingly added to long-horizon agents, usually on the assumption that they help. This paper tests that assumption against the simplest baseline, naive RAG over raw history, and reports that many dedicated systems fall short of it in many cases. It also offers a vocabulary: memory benefit versus memory tax, so that a memory design is judged by its net value and not by standalone performance alone.

Who it affects

Teams building or choosing memory systems for agents, and researchers who design benchmarks for agent memory. The authors say the concept speaks to both system design and evaluation, and they extend it to more application scenarios of memory.

How to use it

The practical idea is to use naive RAG over raw history as the reference when assessing any memory system, and to ask whether the system's gains exceed the evidence it removes or obscures. The authors say they will open-source code and results; a project page is at https://hexuan21.github.io/MemoryTax/.

How solid is it

The claims come from the authors' own abstract as listed on Hugging Face papers. The abstract gives no numeric results and does not name the memory systems, benchmarks or backbones, so the size of the effect cannot be judged from it. The wording is "many" systems underperform "in many cases", not all. Code and results are promised, not yet confirmed as released.

Risks and caveats

The finding is about many systems in many cases, not a verdict that memory never helps. The authors themselves note that deviations from raw history are double-edged and are designed to benefit the task. The abstract does not say which systems lose or by how much, so readers should not assume any particular memory product is affected until the full paper and released results are checked.

“We hope the notion of memory tax can provide a new lens and shift more attention from standalone performance to the net value of memory, informing more reliable memory designs and evaluations.”

— From the paper's abstract