FPD, a new method for unlearning private data from multimodal LLMs, perturbs FFN activations
Multimodal Large Language Models (MLLMs) can memorize and reproduce sensitive information from their training data, which the authors describe as a significant privacy risk. Because these models encode knowledge across both text and images, removing private information selectively is harder still.
Many existing unlearning methods work by training the model toward constructed labels or output distributions. The paper calls these supervision targets. The authors say such methods often face a trade-off between forgetting effectiveness and model utility, and they identify the limitations of the supervision targets as a key factor behind it.
They back this with an empirical observation. Targets that contain more private information lead to insufficient forgetting. Larger distribution shifts away from the model's original output distribution are associated with greater utility degradation. The source says "associated with", not that the shift causes the loss.
Building on these findings, the paper proposes FFN-Perturbed Self-Distillation (FPD). It starts from the teacher's original output distribution, treated as a zero-shift reference, and selectively perturbs the feed-forward network (FFN) activations associated with private information, across both textual and visual modalities. By reducing private information in that original distribution, FPD builds supervision targets that carry less private information and involve less distribution shift.
The authors report that extensive experiments on S-MLLMUn Bench show FPD achieves a better forgetting-utility trade-off than existing methods. The text gives no scores, margins, model names or list of baselines.
Key facts
- The paper proposes FFN-Perturbed Self-Distillation (FPD) for unlearning private information from multimodal LLMs.
- It argues that the supervision targets used by existing unlearning methods drive the trade-off between forgetting and model utility.
- Empirically, targets with more private information give insufficient forgetting; larger shifts from the original output distribution are associated with greater utility degradation.
- FPD starts from the teacher's original output distribution and perturbs FFN activations tied to private information in both textual and visual modalities.
- On S-MLLMUn Bench, the authors report a better forgetting-utility trade-off than existing methods.
Why it matters
Models that memorize and can reproduce sensitive training data are a privacy risk, and multimodal models complicate removal because knowledge is encoded in both text and images. The paper's contribution is a diagnosis as much as a method: it points at the design of the supervision targets as a main source of the familiar trade-off between forgetting and keeping the model useful.
Who it affects
Researchers working on machine unlearning and privacy in multimodal LLMs are the direct audience. Anyone who trains or deploys MLLMs on data that may include personal information has a stake in whether such removal can be done without wrecking the model.
How to use it
This is a research proposal, not a product. The method, FPD, builds supervision targets by perturbing FFN activations linked to private information in the teacher's original output distribution, then trains toward them. The source text mentions no code or data release.
How solid is it
The claims come from the authors' own abstract: experiments on S-MLLMUn Bench show a better forgetting-utility trade-off than existing methods. No numbers, baselines or model names are given in the text, so the size of the improvement cannot be judged from it. It is a preprint on arXiv.
Risks and caveats
The link between distribution shift and utility loss is described only as an association. The source does not claim FPD fully removes private information, only that it gives a better trade-off. Results are reported on one benchmark, S-MLLMUn Bench, and the details of the comparison are not in the text.