Study maps pre-attention spikes and plateaus in hybrid linear attention LLMs

A new paper presents what its authors call the first systematic study of massive activations (MAs) in hybrid linear attention (HLA) LLMs, models that interleave linear attention layers with full attention layers. The study finds that MAs in this architecture follow two consistent, architecture-aligned patterns rather than showing up as scattered anomalies.

The first pattern is what the authors call pre-attention spikes (PAS): massive activations consistently spike immediately before full attention layers. The second is inter-spike plateaus (ISP): the activations can persist through the linear attention layers sitting between one full attention layer and the next, instead of dying out across the gap. The denser a model's full attention layers become, the more its pre-attention spikes connect up through inter-spike plateaus, until the pattern recovers the stable MA morphology already known from full attention LLMs.

The authors establish that this PAS/ISP organization recurs across five linear attention architectures, six hybridization configurations, five data domains, and representative open-source hybrid models ranging from 1.2B to 397B total parameters. The abstract does not name which specific architectures, configurations, domains or models were used.

To probe the cause, the authors also ran controlled pretraining of GDN-based hybrid models at scales up to 1.3B total parameters. Both PAS and ISP emerge early in training and respond asymmetrically to output gating: full attention output gating strongly attenuates the magnitude of the activations without eliminating their layerwise organization, while removing GDN gates produces only a comparatively modest amplification. The paper describes both effects only qualitatively, without exact numeric magnitudes.

Mechanistically, the authors' systematic-outlier analysis supports what they call a shared lifecycle account, in which both patterns are governed by the timing of when the massive activations get cancelled. PAS is described as following a localized write-sink-cancel process, while the extended persistence of ISP is consistent with delayed cancellation. At the limit where a model is built entirely of full attention, this account recovers the stable MA morphology already known for full attention LLMs.

The code behind the analysis is public at https://github.com/StartluxLabs/Massive-Activations-HLA. The abstract does not state a publication venue or peer-review status, and it does not compare its findings to prior work beyond describing itself as the first systematic study of the phenomenon.

Key facts

The paper presents what it calls the first systematic study of massive activations (MAs) in hybrid linear attention LLMs, identifying two consistent patterns: pre-attention spikes (PAS), where MAs spike immediately before full attention layers, and inter-spike plateaus (ISP), where MAs persist through the linear attention layers in between.
The PAS/ISP organization recurs across five linear attention architectures, six hybridization configurations, five data domains, and open-source hybrid models ranging from 1.2B to 397B total parameters, though the abstract does not name the specific architectures or models tested.
As a model's full attention layers become denser, successive PAS become increasingly connected through ISP, and the pattern recovers the stable MA morphology already known from full attention LLMs.
Controlled pretraining of GDN-based hybrids at scales up to 1.3B total parameters shows both patterns emerge early and respond asymmetrically to gating: full attention output gating strongly attenuates their magnitude without eliminating the pattern, while removing GDN gates produces only modest amplification.
The authors propose a mechanistic lifecycle account in which PAS follows a localized write-sink-cancel process and ISP reflects delayed cancellation; the analysis code is public on GitHub.

Why it matters

This paper is presented as the first systematic study of massive activations in hybrid linear attention LLMs, a class of models that interleave linear attention layers with full attention layers. Instead of turning up scattered, one-off anomalies, the authors describe a consistent, architecture-aligned organization: activations spike immediately before full attention layers (PAS) and can hold steady across the linear attention layers in between (ISP). As full attention becomes denser in a given architecture, the two patterns increasingly merge and the whole thing recovers the stable MA morphology already documented in full attention LLMs. On top of that description, the paper offers a mechanistic account tying both patterns to the timing of when the underlying activations get cancelled, which moves the finding from an observation to an explanation.

Who it affects

The paper does not name specific companies, teams, or model families. Its claims are said to recur across five linear attention architectures, six hybridization configurations, five data domains, and "representative open-source hybrid models" spanning 1.2B to 397B total parameters, but the abstract does not identify which ones. In practice, the work speaks to researchers and engineers who design, train, or study hybrid linear attention LLMs: it gives them a documented, cross-architecture account of where massive activations show up, plus, from the controlled pretraining experiments, evidence that different gating choices, full attention output gating versus removing GDN gates, change the size of those activations without changing where in the network they occur.

How to use it

There's no product or price to weigh here. The practical hook is the released code, at https://github.com/StartluxLabs/Massive-Activations-HLA, which lets researchers reproduce the activation-morphology analysis or run the same PAS and ISP diagnostics against other hybrid linear attention models of their own.

How solid is it

The core finding rests on a broad survey: the PAS/ISP organization is reported across five linear attention architectures, six hybridization configurations, five data domains, and open-source hybrid models spanning 1.2B to 397B total parameters, so it is not a single-model artifact. The gating claims rest on a narrower, controlled setting, pretraining GDN-based hybrids from scratch at scales up to 1.3B total parameters, well below the 397B-parameter ceiling of the broader survey, so the causal story about gating has been tested at a smaller scale than the morphology itself. Within that controlled setting the paper reports only the direction of the effects, that full attention output gating strongly attenuates magnitude and that removing GDN gates brings comparatively modest amplification, without exact numeric magnitudes for either. The source also does not state a publication venue or peer-review status.

Risks and caveats

The abstract does not name any of the five linear attention architectures, six hybridization configurations, five data domains, or the specific open-source hybrid models it tested, so readers cannot check whether a model they use or care about was among them. It also does not compare its findings to, or cite, prior work beyond describing itself as the first systematic study of the phenomenon, so how this account relates to earlier explanations of massive activations in standard transformers is left unaddressed.

“MAs consistently spike immediately before full attention layers, forming pre-attention spikes (PAS), and can persist through intervening linear attention layers, giving rise to inter-spike plateaus (ISP).”

— the authors