Glonce

Muir linear attention reports 1.8-2.0% lower perplexity than GDN-2

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact42Novelty55Relevance100Surprise25Players10Credibility75
Muir linear attention reports 1.8-2.0% lower perplexity than GDN-2

Linear attention squeezes context into a fixed-size key-value state, which gives linear-time training and constant-memory inference. Gated delta-rule models decay that state and erase its prediction for an incoming key before writing a new value. Gated DeltaNet-2 (GDN-2) generalizes the idea with channel-wise decay and decoupled, full-rank erase and write gates.

The Muir paper starts from an analysis of that design. The authors show that state updates are far less sensitive to erasure than to writes. Key activation and unit normalization concentrate key energy onto a sparse subset of channels, whereas value writes scale all channels, so the erase gate can be parameterized at lower rank. They also find that slow-decaying state rows accumulate much larger norms than fast rows. Those rows dominate unweighted readout and force stronger write-time erasure to suppress interference.

Guided by this, they introduce Muir. It fuses all gates into a sensitivity-allocated low-rank projection, re-weights state rows with a matched-filter readout gate, and reinvests the saved parameters in a wider feed-forward network. Because standard chunkwise algorithms become numerically unstable at the large chunk sizes needed to saturate wide matrix units, the authors also design a parallel update for stable, efficient training.

Results, as reported in the abstract: across 166M-1.34B parameters, Muir lowers perplexity by 1.8-2.0% over GDN-2 and by 2.5-3.6% over a Transformer baseline. It raises commonsense accuracy by up to 1.9 points and improves extractive recall over GDN-2. At 1.34B, it outperforms both baselines on multi-hop and multi-needle recall, extrapolates 2,560x from its 4K training length to 10M-token streams without perplexity drift, and keeps near-100% passkey retrieval to 1M tokens with static readout.

On speed, the authors say Muir matches GDN-2 throughput, reaches its 100B-token validation loss 1.2x faster, and trains 2.7x faster than a Transformer at 32K.

Key facts

  • Muir fuses all gates into a sensitivity-allocated low-rank projection, adds a matched-filter readout gate, and moves the saved parameters into a wider feed-forward network.
  • Across 166M-1.34B parameters, perplexity falls 1.8-2.0% versus Gated DeltaNet-2 and 2.5-3.6% versus a Transformer; commonsense accuracy rises by up to 1.9 points.
  • At 1.34B, Muir extrapolates 2,560x, from a 4K training length to 10M-token streams, without perplexity drift, and holds near-100% passkey retrieval to 1M tokens with static readout.
  • The authors report throughput matching GDN-2 and training 2.7x faster than a Transformer at 32K.
  • A new parallel update is meant to keep large-chunk training numerically stable.

Why it matters

Linear attention is attractive because training cost grows linearly with length and inference memory stays constant, but the models usually have to prove they can match Transformers on quality. Muir's pitch is that a closer look at which gates really matter lets you shrink them and spend the savings elsewhere. The finding behind it is that erasure in the state update is far less sensitive than writing, so the erase gate does not need full rank. The abstract reports that the result beats both GDN-2 and a Transformer baseline on perplexity across the 166M-1.34B range, while keeping GDN-2's throughput.

Who it affects

Mainly researchers and engineers working on efficient sequence models, especially gated delta-rule and other linear-attention architectures, and anyone who needs very long context at constant inference memory. The abstract names no authors or institutions.

How to use it

There is nothing to download or deploy from the abstract: no release of code, weights or a date of submission is mentioned. For practitioners, the useful part is the design recipe. Allocate gate rank by sensitivity (lower rank for erasure than for writes), weight state rows at readout so slow-decaying rows do not dominate, and spend the parameters saved on a wider feed-forward network. A parallel update is described for stable training with large chunk sizes.

How solid is it

This is a single preprint abstract, and the claims are the authors' own. No independent replication or peer review is mentioned. The perplexity figures are percentage reductions relative to the baselines, not point differences, and the 1.8-2.0% and 2.5-3.6% ranges cover the whole 166M-1.34B span; per-size figures are not given. No absolute perplexity values or benchmark names are given either. The 'up to 1.9 points' commonsense gain is not tied to a specific baseline or benchmark.

Risks and caveats

The headline long-context results are for the 1.34B model only: the 2,560x extrapolation to 10M tokens and the near-100% passkey retrieval to 1M tokens with static readout. Passkey results beyond 1M tokens are not given. The abstract does not say which baseline the '1.2x faster' to reach the 100B-token validation loss is measured against, and it gives no hardware, training-token budget for the larger runs, or wall-clock times. The models are small by current standards, so how the gains hold at larger scale is not shown.