CurvDelta paper builds capability preservation into LLM updates

CurvDelta paper builds capability preservation into LLM updates

The paper starts from a familiar problem in adapting large language models: capability preservation. An adaptation can reach its target while degrading the model's general capabilities.

The authors introduce CurvDelta, which they describe as a scalable framework that builds the preservation constraint into the parameterization of the model update itself. It analyzes the local geometry of the preservation constraint around the pre-adaptation model. From that, it constructs a low-curvature space in which parameter updates leave the preservation loss nearly invariant. To make this construction practical for LLMs, the authors use Kronecker-factored approximate curvature (K-FAC) and a factorized projection.

They also introduce CurvLoRA, a low-rank variant within the same preservation geometry, designed for more efficient adaptation. The authors say both methods apply to preservation-constrained adaptation tasks such as capability learning and selective unlearning.

The evaluation covers three areas: mathematical reasoning, code generation and knowledge unlearning. On math and code, the authors report competitive downstream performance, not better performance, while better preserving general capabilities. They describe this as a favorable acquisition-retention trade-off relative to the baselines. On unlearning, their main results on the TOFU and MUSE benchmarks show a better balance between targeted forgetting and knowledge retention than the baselines. Retention performance is close to the from-scratch retraining reference, not equal to it.

The text available is the paper's abstract. It gives no numerical scores.

Key facts

  • CurvDelta puts the capability-preservation constraint into the parameterization of the LLM update.
  • It builds a low-curvature space around the pre-adaptation model in which updates leave the preservation loss nearly invariant, made practical with K-FAC and a factorized projection.
  • CurvLoRA is a low-rank variant in the same preservation geometry, aimed at more efficient adaptation.
  • On math reasoning and code generation, the authors report competitive downstream performance with better preservation of general capabilities than the baselines.
  • On TOFU and MUSE unlearning, they report a better forgetting-retention balance than the baselines, with retention close to the from-scratch retraining reference.

Why it matters

Fine-tuning a model for one goal can erode what it can do elsewhere. CurvDelta tackles this by restricting where the update can move, using the geometry of the preservation loss. The same idea is applied to two opposite aims: learning new capabilities and selectively forgetting knowledge.

Who it affects

Researchers and engineers who adapt LLMs to new skills while needing the base model's general abilities to hold up. It also concerns anyone working on unlearning, where the goal is to remove specific knowledge while keeping the rest, as measured on TOFU and MUSE.

How to use it

The abstract describes two methods: CurvDelta, and CurvLoRA for more efficient low-rank adaptation. Both target preservation-constrained tasks such as capability learning and selective unlearning. The abstract does not mention a code release, a license or compute cost, so practical adoption details would have to come from the full paper.

How solid is it

This is a preprint listed on Hugging Face Papers, and the claims here come from the authors' own abstract. They are qualitative: competitive performance on math and code, a better forgetting-retention balance on TOFU and MUSE, and retention close to the retraining reference. No accuracy, forgetting or retention scores are given, and the baselines and models used are not named, so the size of any gain cannot be judged from this text.

Risks and caveats

Math and code results are described as competitive, not superior, so the gain is in preservation, not in downstream performance. Unlearning retention is close to, not equal to, the from-scratch retraining reference. Without named baselines, models or numbers, the strength of the evidence is unclear until the full paper and any independent replication are available.

“CurvDelta constructs a low-curvature space in which parameter updates leave the preservation loss nearly invariant.”

— From the paper's abstract