Glonce

Qwen2.5-1.5B relearns a forgotten fact faster, arXiv paper reports

researchThe Glonce editorsarXiv🔥57Glonce rating57Impact42Novelty62Relevance92Surprise35Players8Credibility74

The paper starts from a simple question: when a fact becomes harder to get out of a language model, has the model lost it, or has it only become less accessible? Test-time continual learning agents, which update their parameters as they go, are usually judged by their current output accuracy. The authors ask whether that measure captures how earlier experience shapes later updates.

The specific test is whether learning one factual binding gives a later relearning advantage that goes beyond mere familiarity with the entity, the answer vocabulary, the format or the query slot. The experiments use Qwen2.5-1.5B and compare three matched, counterbalanced histories. LEARNED receives the correct binding. CONTROL receives matched exposure without it. WRONG receives a competing, relation-valid answer in the same slot.

All three then go through an identical continual fine-tuning stream with no rehearsal. At a prespecified checkpoint of partial forgetting, LEARNED reacquires the target faster than CONTROL. WRONG shows no positive advantage over CONTROL. Its comparison with LEARNED gives slot-matched corroboration, though the authors flag that competitor interference may play a part. The advantage stays positive after adjusting for measured behavior before relearning.

In a secondary, descriptive conversion that uses an unpaired pool of never-seen facts, prior learning reduces the optimizer updates needed for restoration by about 18%. The authors conclude that current behavioral accuracy can underestimate the lasting effect of previously acquired experience. Code is available at https://github.com/zhezhou1106/relearning-savings.

Key facts

  • The study uses Qwen2.5-1.5B and compares three matched, counterbalanced histories: LEARNED (correct binding), CONTROL (matched exposure without it) and WRONG (a competing relation-valid answer in the same slot).
  • After an identical rehearsal-free continual fine-tuning stream, LEARNED reacquires the target faster than CONTROL at a prespecified checkpoint of partial forgetting.
  • WRONG shows no positive advantage over CONTROL; the advantage of LEARNED remains positive after adjusting for measured pre-relearning behavior.
  • In a secondary descriptive conversion with an unpaired never-seen pool, prior learning reduces the optimizer updates needed for restoration by about 18%.
  • The authors argue that current behavioral accuracy can underestimate the lasting effect of previously acquired experience; code is on GitHub.

Why it matters

Agents that keep updating their weights at test time are normally scored by what they output right now. This paper argues that a fact which no longer comes out correctly may still leave a trace: the model that had learned it picks it up again faster than a model that never did. If that holds, accuracy alone misses part of what a model has retained. The design also tries to separate a true fact-specific effect from general familiarity with the entity, the answer words, the format or the query slot.

Who it affects

Mainly researchers working on continual learning and test-time learning agents, who evaluate models that update parameters over time. It is also relevant to anyone studying forgetting in language models and how to measure it.

How to use it

The authors link code at https://github.com/zhezhou1106/relearning-savings. The setup is a template for measuring relearning savings: build LEARNED, CONTROL and WRONG histories, run the same rehearsal-free fine-tuning stream, and compare how quickly each reacquires the target at a prespecified point of partial forgetting. It is a measurement protocol, not a product or a method for improving a model.

How solid is it

This is a preprint on arXiv. The primary result is the comparison of LEARNED against CONTROL; the controls are matched and counterbalanced, and the advantage survives adjustment for measured pre-relearning behavior. The WRONG comparison is described as corroboration that is subject to possible competitor interference. The 18% figure is a secondary descriptive conversion from an unpaired never-seen pool, not the primary test. The effect size for the primary comparison is not given. No statistical significance, confidence intervals or number of facts/seeds is given.

Risks and caveats

Only one model, Qwen2.5-1.5B, is mentioned; no larger models or other model families are reported, and no claim is made about larger-scale or production systems. The checkpoint of partial forgetting is called prespecified, but its level is not stated. The 18% reduction should not be read as a paired headline result. The authors' conclusion is worded with 'can', not 'does': accuracy can underestimate the lasting effect of prior experience.

“Current behavioral accuracy can therefore underestimate the lasting effect of previously acquired experience.”

— From the paper's abstract (arXiv 2610.22381)