Glonce

Deep Fusion Memory adds chunk-level parametric memory to frozen LLMs

researchThe Glonce editorsHugging Face🔥58Glonce rating58Impact45Novelty60Relevance95Surprise30Players10Credibility70
Deep Fusion Memory adds chunk-level parametric memory to frozen LLMs

A paper on the Hugging Face papers page introduces Deep Fusion Memory (DFM), a way to give a large language model access to external knowledge without continued pretraining and without retrieval at inference time.

The authors start from three problems. Continued pretraining risks catastrophic forgetting. Retrieval at inference time can add substantial overhead. External parametric memory avoids online retrieval, but building its training data requires extensive offline search, and output-level interpolation provides only shallow fusion.

DFM is a parametric memory architecture meant to reduce that construction workload and to put learned memory into the model's intermediate hidden states rather than blending it in at the output. A causal memory predictor learns to generate memory from the preceding context, using retrieved chunk embeddings from the corpus as supervision. The predicted memory is shared across all tokens within each chunk and fused into their hidden states through gated cross-attention. Memory prediction and fusion are trained end to end while the backbone stays frozen, and inference needs no explicit retrieval.

On cost, the abstract compares DFM with existing parametric memory approaches, which often construct token-level memory targets. DFM needs 32 times fewer retrieval queries and 2048 times fewer retrieved items to construct its training data.

On quality, across six QA tasks DFM beats the strongest baseline by 12.71% on Mistral-7B-v0.3 and by 11.19% on Qwen3-8B-Base. Beyond QA, domain adaptation gives average gains of 6.51 points over Qwen3-4B-Base and 4.38 points over Qwen3-8B-Base.

Key facts

  • DFM predicts chunk-level memory from the preceding context and fuses it into hidden states through gated cross-attention, with the backbone frozen.
  • Inference requires no explicit retrieval; retrieved chunk embeddings are used only as supervision during training.
  • Training-data construction needs 32 times fewer retrieval queries and 2048 times fewer retrieved items than token-level parametric memory approaches.
  • Across six QA tasks, DFM beats the strongest baseline by 12.71% on Mistral-7B-v0.3 and 11.19% on Qwen3-8B-Base.
  • Domain adaptation yields average gains of 6.51 points on Qwen3-4B-Base and 4.38 points on Qwen3-8B-Base.

Why it matters

Teaching a model new knowledge usually means a trade-off. Continued pretraining risks catastrophic forgetting, and retrieval at inference time can add substantial overhead. Existing parametric memory sidesteps online retrieval but is costly to build and fuses only shallowly at the output. DFM targets both weak points: it predicts memory per chunk rather than per token, which cuts the retrieval work behind its training data, and it injects that memory into intermediate hidden states.

Who it affects

Researchers working on knowledge injection, retrieval-augmented generation and parametric memory are the direct audience. Teams that want a frozen open model, such as the Mistral-7B-v0.3 and Qwen3 base models used here, to handle question answering or a specific domain without retrieval at serving time may find the approach relevant.

How to use it

The abstract describes a training recipe rather than a product. A causal memory predictor is trained with retrieved chunk embeddings as supervision, and a gated cross-attention module fuses the predicted memory into the hidden states of a frozen backbone. Both parts are trained end to end. At inference the model runs without explicit retrieval. The abstract does not state a chunk size.

How solid is it

The evidence is the abstract's own reported results: six QA tasks on Mistral-7B-v0.3 and Qwen3-8B-Base, plus domain adaptation on Qwen3-4B-Base and Qwen3-8B-Base. The figures are the authors' claims. The abstract does not say whether the 12.71% and 11.19% gains are absolute or relative, or whether they are averaged over the tasks. It also does not name the six QA tasks, the baselines or the adaptation domains.

Risks and caveats

The abstract gives no inference latency or memory-cost figures, so the practical saving at serving time is not quantified; the 32 times and 2048 times figures concern training-data construction only. No code, model or dataset release is mentioned. QA results are reported for Mistral-7B-v0.3 and Qwen3-8B-Base, and domain-adaptation gains for Qwen3-4B-Base and Qwen3-8B-Base, so the abstract does not show every result on every model.