Glonce

Sliding window attention beats linear attention, study finds

researchThe Glonce editorsHugging Face🔥75Glonce rating75Impact68Novelty72Relevance95Surprise72Players40Credibility82
Sliding window attention beats linear attention, study finds

A paper argues that a much simpler technique beats the fashionable fix for the cost of quadratic attention. Standard attention gets more expensive with every new token, because the keys and values for every past token have to stay in memory indefinitely, which the authors call unsustainable at scale. One proposed fix has been retrofitting existing LLMs to use linear attention instead, a post-training conversion pitched as reaching state-of-the-art performance at much lower cost. The authors note this line of work has attracted a lot of interest but had not been properly checked against simple baselines. They ran that check. Across multiple LLMs and various downstream tasks, they find that Sliding Window Attention (SWA) with attention sinks performs as well as or better than models that were post-trained into linear attention. The gap is largest on long-context reasoning: on the Needle-in-a-Haystack and BABILong benchmarks, SWA scores 2 to 10 times higher than the linear-attention models. SWA also needs no post-training at all, is described as extremely fast, and keeps memory use low. The authors' practical conclusion is to switch to SWA rather than post-training a model into linear attention to cut inference memory costs; they add that linear attention may still hold promise, but likely only if trained from scratch or given extensive post-training, since that is what it would take to even match SWA.

Key facts

  • Sliding Window Attention (SWA) with sinks matches or beats post-trained Linear Attention models across multiple LLMs on various downstream tasks.
  • On long-context reasoning benchmarks Needle-in-a-Haystack and BABILong, SWA scores 2 to 10 times higher than post-trained linear attention.
  • SWA needs no post-training, is extremely fast, and keeps memory use low, unlike a linear-attention retrofit.
  • The authors say linear attention likely needs training from scratch or extensive post-training just to match SWA.
  • The authors say prior work retrofitting LLMs with linear attention had not been properly compared against simple baselines like SWA.

Why it matters

Quadratic attention forces every additional token's keys and values to be kept in memory indefinitely, which the authors call unsustainable as context grows. Linear attention retrofits were proposed as the fix, promising state-of-the-art results at much lower cost, and drew significant interest on that promise. This paper's contribution is a baseline check that line of work reportedly skipped: it shows a far simpler method, sliding window attention with sinks, already matches or beats post-trained linear attention, and does so without any retraining.

Who it affects

Anyone deciding how to cut the inference memory cost of a long-context LLM, including teams weighing whether to post-train a model into linear attention. The authors' recommendation is aimed directly at that decision: use SWA instead. It also bears on researchers evaluating linear-attention methods, since the paper's central claim is that those methods have not been benchmarked against a strong enough baseline.

How to use it

SWA with sinks can be applied without any post-training, so it is available immediately at inference time. The authors describe it as extremely fast and low in memory use. By contrast, matching SWA's performance with linear attention would likely require training a model from scratch or extensive post-training, according to the authors, which is a much larger investment for a method this paper finds does not currently outperform the simpler alternative.

How solid is it

The comparison spans multiple LLMs and various downstream tasks, with a quantified result on two specific long-context reasoning benchmarks, Needle-in-a-Haystack and BABILong, where SWA scores 2 to 10 times higher. The source text does not name which LLMs or which downstream tasks were used, and gives no numeric scores for the broader downstream-task comparison beyond the qualitative claim that SWA performs as well or better.

Risks and caveats

The specific models and downstream tasks behind the headline comparison are not disclosed in the available text, which limits independent checking of the result. The comparison is specifically against post-trained linear attention models; the authors themselves flag that linear attention trained from scratch or given extensive post-training might close the gap, so the finding is a statement about the retrofit approach rather than a final verdict on linear attention as a concept.

“we strongly recommend switching to SWA instead of post-training linear models”

— the authors