MERIT benchmark finds memory implementation beats presence in AI agents

Researchers have built MERIT (Memory Evaluation for Realistic Instrumented Tasks), a benchmark and test harness designed to measure whether long-term memory changes what a tool-using AI agent actually does, rather than just whether it can answer questions about past conversation. Existing memory benchmarks such as LoCoMo and LongMemEval test conversational recall: they ask an agent questions about dialogue history. MERIT instead gives agents episodic tool-use tasks across three domains, where completing the task depends on facts learned in an earlier episode; an automated leak check verifies that this dependency is real rather than solvable by guessing. A difficulty ladder built into the tasks ends in cases that require recalling a fact that has since been updated, and the benchmark also introduces controlled memory corruption to stress-test retrieval. Every memory operation is metered in tokens and dollars, so cost, not just accuracy, is part of the score.

The team ran 23,440 scored episodes for a total cost of $42.57, combining a two-generation pilot on gpt-4.1-mini with a preregistered grid of three models and three seeds (GPT-4.1 and Claude Haiku 4.5, with the memory implementation held fixed across models) and a spot-check on Claude Sonnet 5 gated on a clean full-replay control. Without memory, dependent-task success sits at a leak-verified floor of 0.00; with memory in place, it rises to between 0.55 and 1.00. But on tasks that require recalling an updated fact, embedding-based retrieval collapses unpredictably, scoring between 0.30 and 0.95 across models, with a gap of up to 0.45 between different random seeds of the same setup, and even when the agent retrieves the correct value it acts on it only 55% of the time. Update-on-write memory stores, meaning a structured fact store and, notably, LLM summarization, stay far more reliable, scoring between 0.70 and 1.00. A hybrid method that combines approaches performs worse than the fact store used on its own. The Claude Sonnet 5 spot-check reproduces the same pattern seen in the main grid. Overall, swapping one memory implementation for another can move task success by up to 60 points, and replaying the full history is never the economical choice: the best-performing memory condition in each domain delivers 2.7 to 3.9 times the marginal utility per dollar that full replay does. The researchers say they are releasing the benchmark, the harness, and all traces from the runs.

Key facts

MERIT is a new benchmark that tests whether AI agent memory changes task outcomes, unlike prior benchmarks (LoCoMo, LongMemEval) that only test conversational recall.
The evaluation ran 23,440 scored episodes for $42.57, using gpt-4.1-mini in a pilot plus a preregistered grid of GPT-4.1 and Claude Haiku 4.5, and a Claude Sonnet 5 spot-check.
Memory lifts dependent-task success from a leak-verified floor of 0.00 to between 0.55 and 1.00.
On updated facts, embedding retrieval collapses unpredictably (0.30-0.95 across models, with a seed-to-seed gap of up to 0.45), and agents act on a correctly retrieved value only 55% of the time.
Update-on-write stores (a structured fact store and LLM summarization) hold at 0.70-1.00 and beat a hybrid approach; full replay is never economical, with the best condition delivering 2.7-3.9x the marginal utility per dollar.

Why it matters

Memory benchmarks today, such as LoCoMo and LongMemEval, measure whether an agent can answer questions about a past conversation. They do not measure whether remembered facts change what a task-executing agent actually does, or whether remembering is worth what it costs. MERIT is built to close that gap: it scores memory by its effect on task completion in tool-use settings, under explicit token and dollar accounting, rather than by recall accuracy alone.

Who it affects

The direct audience is researchers and engineers building or evaluating long-term memory systems for tool-using LLM agents. By extension, it matters to anyone deploying agents that are expected to remember facts across sessions, since the results show that the choice of memory implementation, not just its presence, decides whether that memory helps or quietly fails.

How to use it

The researchers say they are releasing the benchmark, the harness, and all traces from the runs, so teams can score their own agents on the same episodic tool-use tasks and cost metering rather than on conversational recall alone. The results themselves point to a concrete choice: update-on-write approaches, a structured fact store or LLM summarization, held up far better on updated facts (0.70-1.00) than embedding-based retrieval (0.30-0.95, with results varying widely by seed), and combining approaches in a hybrid setup performed worse than the fact store alone.

How solid is it

The evaluation is large and cost-tracked: 23,440 scored episodes across a two-generation pilot and a preregistered three-model, three-seed grid, with a separate spot-check on the newer Claude Sonnet 5 gated on a clean full-replay control. The dependent-task setup is leak-verified, meaning the researchers automatically confirmed that success genuinely requires the earlier-episode fact rather than being solvable without it, and the zero-memory floor came out at exactly 0.00, matching that design.

Risks and caveats

The source text does not name individual authors or institutions, does not describe the three task domains beyond calling them 'three domains,' and does not spell out what the difficulty ladder or the controlled memory corruption concretely involve. It also does not give a publication date or say where the released materials will be hosted. On the results themselves, the seed-to-seed gap of up to 0.45 on updated facts, and the finding that agents act on a correctly retrieved value only 55% of the time, both suggest that reported memory performance for a single run can be an unreliable guide to how a system behaves in general.