Glonce

Deletion Tuning matches full fine-tuning by zeroing under 0.05% of LLM weights

researchThe Glonce editorsHugging Face🔥66Glonce rating66Impact50Novelty70Relevance100Surprise55Players10Credibility60
Deletion Tuning matches full fine-tuning by zeroing under 0.05% of LLM weights

A paper listed on Hugging Face Papers asks how little of a pretrained large language model has to change for it to acquire new capabilities. Its answer: very little. The authors report that zeroing fewer than 0.05% of the model's weights, with no other modification, matches full fine-tuning (FFT) at scale.

The result is claimed across three post-training regimes: reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. The method is called Deletion Tuning. It learns a binary mask over the gate projection of transformer blocks, so training amounts to deciding which weights to delete rather than adjusting their values.

The authors name two practical advantages. First, the trained mask is hundreds to thousands of times smaller than the corresponding FFT checkpoint. Second, Deletion Tuning forgets far less, which the authors say addresses a central challenge in continual learning. In the reinforcement learning setting, its prior-task accuracy stays near the base level, while FFT, LoRA and random sparse fine-tuning lose up to 5.4 points.

To explain why masking alone is enough, the authors analyze the geometry of the weight changes learned under reinforcement learning. They use three measures: stable rank, spectrum drift and principal-weight overlap. Compared with LoRA and random sparse fine-tuning, Deletion Tuning comes closest to FFT on all three.

Key facts

  • Zeroing fewer than 0.05% of a pretrained LLM's weights, with no other change, is reported to match full fine-tuning at scale.
  • The match is claimed across three regimes: reinforcement learning with verifiable rewards, supervised fine-tuning and on-policy distillation.
  • Deletion Tuning learns a binary mask over the gate projection of transformer blocks; the mask is hundreds to thousands of times smaller than the FFT checkpoint.
  • Under RL, Deletion Tuning's prior-task accuracy stays near the base level, while FFT, LoRA and random sparse fine-tuning lose up to 5.4 points.
  • Analysis with stable rank, spectrum drift and principal-weight overlap finds Deletion Tuning closest to FFT, compared with LoRA and random sparse fine-tuning.

Why it matters

Post-training usually means changing many weights and storing a full new checkpoint. This paper claims that deleting a tiny fraction of weights, fewer than 0.05%, can do the same job as full fine-tuning in three different regimes. If it holds, it reframes how much of a model actually has to change to learn something new. The reported reduction in forgetting speaks to continual learning, which the authors call a central challenge.

Who it affects

The claim is aimed at anyone who post-trains LLMs with reinforcement learning, supervised fine-tuning or on-policy distillation, and at people working on continual learning, where learning a new task erodes old ones. The abstract also points to storage: a mask hundreds to thousands of times smaller than an FFT checkpoint.

How to use it

The abstract describes the recipe in one line: learn a binary mask over the gate projection of transformer blocks and leave everything else untouched. No code or checkpoint release is mentioned, so there is nothing to download from what is available. Practitioners would need the full paper to reproduce it.

How solid is it

Everything here comes from the authors' own abstract, so these are their claims, not independent results. The abstract gives no accuracy figures for the match with FFT, only that it matches. It names no models, model sizes, benchmarks or datasets, so "at scale" cannot be checked from this text. The 5.4-point figure is a maximum loss in prior-task accuracy under RL, and the abstract does not say which of FFT, LoRA or random sparse fine-tuning lost the most.

Risks and caveats

Only the gate projection of transformer blocks is named as the place the mask is applied. The abstract makes no claim about compute or training cost savings, so the gains described are in checkpoint size and forgetting, not in training expense. The analysis of why masking works covers reinforcement learning only. Without model names or benchmarks, how far the result generalizes is unknown.

“How little of a pretrained large language model (LLM) has to change for it to acquire new capabilities?”

— Paper abstract, Hugging Face Papers