Glonce

Model merging quietly erases Gemma's safety classifier, study finds

researchThe Glonce editorsarXiv🔥71Glonce rating71Impact62Novelty70Relevance95Surprise57Players38Credibility85

Researchers ran a controlled case study on model merging, the practice of combining separately fine-tuned models into one without further training, to test whether it preserves more than one safety-relevant behavior at once. They took two versions of Gemma-3-1B-IT, each fine-tuned on a different safety objective: one on CARES harm-level classification (judging how harmful a piece of content is) and the other on WildJailbreak adversarial refusal (resisting jailbreak attempts). They merged the two fine-tunes using four standard methods: Linear, SLERP, TIES, and DARE-TIES, then measured the merged models on classification accuracy, resistance to jailbreak attacks, and compliance with benign requests.

The result was the same across all four merge methods: refusal behavior transferred far better than classification skill. The merged models kept 81 to 85 percent of the jailbreak refusal rate, but CARES classification accuracy fell to at most 12.9 percent. In other words, the merge preserved the model's ability to say no to attacks while destroying its ability to correctly judge how harmful content is.

To find the cause, the researchers measured the weight-space task vectors, the direction and size of the changes each fine-tune made to the model. The two task vectors were nearly orthogonal, with a cosine similarity of just 0.011, meaning the two fine-tunes were not pulling the model in directly opposing directions. Instead, the refusal fine-tune consistently produced larger per-layer task-vector magnitudes than the classification fine-tune. Because merge methods that are sensitive to vector magnitude effectively weight the larger update more heavily, they systematically favored the refusal behavior over the classification behavior, even though the two updates did not conflict directionally.

The authors conclude that standard model merging can collapse safety recognition into broad refusal whenever the safety-relevant task vectors involved differ substantially in scale, a failure mode that would not show up if only aggregate or single-task performance were checked after merging.

Key facts

  • Two Gemma-3-1B-IT models were separately fine-tuned on CARES harm-level classification and WildJailbreak adversarial refusal, then merged with four methods: Linear, SLERP, TIES, and DARE-TIES.
  • Across all four methods, merged models kept 81-85% jailbreak refusal rates while CARES classification accuracy fell to at most 12.9%.
  • The two task vectors were nearly orthogonal (cosine similarity 0.011), ruling out direct directional conflict as the cause of the asymmetry.
  • The refusal fine-tune consistently produced larger per-layer task-vector magnitudes, so magnitude-sensitive merge methods favored refusal updates over classification updates.
  • The authors conclude standard merging can collapse safety recognition into broad refusal when safety-relevant task vectors differ substantially in scale.

Why it matters

Model merging is a common shortcut for combining capabilities from separately fine-tuned models without paying for further training. This study shows that when two of those capabilities are both safety-relevant, merging does not combine them evenly: a team could merge a harm classifier into a refusal-tuned model and end up with a model that still refuses jailbreaks but has effectively lost its ability to correctly classify harmful content, without any obvious signal that this happened.

Who it affects

Anyone building or deploying models by merging separate safety fine-tunes, including teams that combine a content classifier with a refusal-tuned model, providers of merge tooling, and downstream users who assume a merged model inherits the safety properties of all its ingredients.

How to use it

Before shipping a merged model, test each safety-relevant capability separately rather than trusting aggregate benchmarks. The finding held across four different merge methods (Linear, SLERP, TIES, DARE-TIES), so switching merge algorithms is not a reliable fix; checking the relative magnitude of each fine-tune's weight changes before merging is a more direct diagnostic.

How solid is it

The result is consistent across all four merge methods tested and is backed by a direct weight-space measurement (task-vector cosine similarity and per-layer magnitude) that identifies a specific mechanism rather than just reporting an effect. It is a controlled, single-variable case study.

Risks and caveats

The study covers one small model, Gemma-3-1B-IT, and one specific pair of safety tasks, CARES classification and WildJailbreak refusal. It does not establish how the effect scales to larger models, other model families, or other pairs of capabilities, and it does not test merge methods designed to correct for magnitude imbalance.

“These results show that standard model merging can collapse safety recognition into broad refusal when safety-relevant task vectors differ substantially in scale.”

— the paper's abstract