Deleting 'stale' LLM memories can hurt accuracy, paper finds
LLM memory systems often delete facts that a resolver has classified as stale. A new arXiv paper argues that this hard-filtering step can reduce downstream accuracy when resolver errors delete query-critical facts, meaning facts the question actually depends on.
The authors ran a controlled sensitivity intervention across three models. At perfect resolution, hard-drop gave gains of +0.186 to +0.230. At the lowest tested resolver reliability, the same policy produced losses of -0.176 to -0.181. So the sign of the effect flipped as the resolver got less reliable.
Paired targeted interventions point at the mechanism. According to the authors, a false-drop (deleting a fact that should have been kept) causes significantly greater degradation than a false-keep (retaining a fact that was stale). They also report that a non-deleting baseline can still fail when incorrect labels steer generation, so simply keeping everything and attaching labels is not automatically safe.
On a frozen 102/102 calibration/test split, a simple deployment-matched gate reached 0.655 accuracy. Always hard-drop reached 0.641, and test-informed oracle selection reached 0.662. The gate therefore sits 0.014 above always hard-drop and 0.007 below the oracle.
The authors add a warning about reading their crossover points as settings to copy. Global-accuracy crossovers require gold labels and are model- and task-specific diagnostics, not deployment thresholds. Their recommendation is that deletion should be treated as cost-sensitive: validate query-critical false-drop risk on matched data, or default to label-free retention when such validation is unavailable.
Key facts
- Across three models, hard-drop moves from gains of +0.186 to +0.230 at perfect resolution to losses of -0.176 to -0.181 at the lowest tested resolver reliability.
- Paired targeted interventions show false-drop causes significantly greater degradation than false-keep.
- A non-deleting baseline can still fail when incorrect labels steer generation.
- On a frozen 102/102 calibration/test split, a simple deployment-matched gate scores 0.655, against 0.641 for always hard-drop and 0.662 for test-informed oracle selection.
- The authors say deletion should be cost-sensitive: validate query-critical false-drop risk on matched data, or default to label-free retention.
Why it matters
Deleting facts judged stale is a common step in LLM memory systems, and it looks like a safe way to keep memory clean. This paper argues the step has a hidden cost: when the resolver that decides what is stale makes mistakes, it can delete facts the query needs, and the damage can outweigh the benefit. In the authors' controlled experiment the same policy that gained +0.186 to +0.230 with a perfect resolver lost -0.176 to -0.181 at the lowest tested reliability.
Who it affects
Teams that build memory layers for LLM agents and assistants, especially those that use a resolver or classifier to mark facts as stale and then remove them. It also concerns anyone evaluating such systems, since the authors say global-accuracy crossovers are model- and task-specific diagnostics.
How to use it
The authors' advice is to treat deletion as cost-sensitive. Validate query-critical false-drop risk on data that matches the deployment. If that validation is not available, default to label-free retention. On their frozen 102/102 calibration/test split, a simple deployment-matched gate scored 0.655, slightly above always hard-drop at 0.641.
How solid is it
This is an arXiv preprint, and the account here rests on its abstract. The headline results come from a controlled sensitivity intervention across three models and from paired targeted interventions on false-drop versus false-keep. The gate comparison uses a frozen split of 102 calibration and 102 test items, and the three accuracy figures (0.655, 0.641, 0.662) are single values. The gap between the gate and always hard-drop is 0.014. The authors themselves say the crossovers are model- and task-specific.
Risks and caveats
The authors state that the crossover points require gold labels and are not deployment thresholds, so they should not be copied as settings. A non-deleting baseline is not a guaranteed fix either, because incorrect labels can still steer generation. The gate's lead over always hard-drop is small and was measured on a single 102/102 split, and the oracle that picked using the test split scored slightly higher at 0.662.
“Deletion should instead be treated as cost-sensitive: validate query-critical false-drop risk on matched data, or default to label-free retention when such validation is unavailable.”
— From the paper's abstract