FlashMorph2 picks which layers keep full attention in hybrid models

FlashMorph2 picks which layers keep full attention in hybrid models

Hybrid attention models mix full attention with linear attention to get a better balance of quality and efficiency. Turning a pretrained Transformer into one requires deciding which layers keep full attention under a set budget. A new paper tackles that layer-selection step and introduces FlashMorph2.

The authors start from an existing method, FlashMorph, and identify two off-policy mismatches in how it selects layers. First, it relies on continuous branch interpolation and teacher-forced hidden states, which differ from the discrete hybrid computation the final model actually performs. Second, it keeps the model weights frozen during selection, so it fails to account for how parameters adapt during the recovery phase that follows.

FlashMorph2 is described as an on-policy layer-selection method that fixes both problems. It evaluates sampled discrete hybrid architectures end to end, using each architecture's own hidden states. It also jointly optimizes the layer-selection scores and the model weights under the recovery objective, so the scores reflect performance after adaptation rather than before it.

The experiments cover several Qwen3 model scales and linear-attention variants. The authors report that FlashMorph2 consistently outperforms existing layer-selection methods. Against FlashMorph, average scores improve by +2.42 points on RULER, +1.08 on LongBench and +0.76 on recall-intensive tasks. Layer selection finishes in 0.72 to 3.57 GPU hours across models from 0.6B to 8B parameters, which the authors cite as evidence of high efficiency and scalability.

Key facts

  • FlashMorph2 is an on-policy method for choosing which layers keep full attention when a pretrained Transformer is converted into a hybrid of full and linear attention.
  • The paper names two off-policy mismatches in FlashMorph: continuous branch interpolation with teacher-forced hidden states, and frozen weights that ignore adaptation during recovery.
  • FlashMorph2 evaluates sampled discrete hybrid architectures end to end and jointly optimizes selection scores and model weights under the recovery objective.
  • Reported gains over FlashMorph in average score: +2.42 points on RULER, +1.08 on LongBench, +0.76 on recall-intensive tasks, across Qwen3 scales and linear-attention variants.
  • Layer selection takes 0.72 to 3.57 GPU hours across 0.6B to 8B models.

Why it matters

Hybrid models that blend full and linear attention aim for a better quality-efficiency trade-off than pure Transformers. Converting an existing pretrained model is only as good as the choice of which layers keep full attention. The paper argues that FlashMorph chooses those layers in a way that does not match how the converted model really runs, and offers a selection method that does.

Who it affects

Mainly researchers and engineers who convert pretrained Transformers into hybrid attention models, and anyone building on FlashMorph. The experiments use the Qwen3 family from 0.6B to 8B parameters.

How to use it

The text describes the method and results only. No release of code, weights or a link to a repository is mentioned, so there is nothing to run from this source yet. Practitioners can take the idea: score layers by evaluating sampled discrete hybrids end to end, and tune scores and weights together under the recovery objective.

How solid is it

The numbers are the authors' own claims from the paper's abstract. They are differences against FlashMorph, not absolute scores. The absolute baseline scores on RULER, LongBench and recall-intensive tasks are not given. The existing methods other than FlashMorph are not named, and gains over them are not quantified. The 0.72 to 3.57 GPU-hour range covers model sizes from 0.6B to 8B, and the text does not say which size takes which time.

Risks and caveats

The gains are shown only against one named baseline, and without absolute scores their real size cannot be judged. The full-attention budget, meaning how many layers are kept, is not stated. The linear-attention variants tested are not named, and the GPU type is not stated. Claims of consistent outperformance and scalability rest on the authors' reported experiments, with no independent check in the source.