Physics of Language Models Part 4.2: Canon layers save about 30% of tokens at 8B
The paper, Part 4.2 of the Physics of Language Models series, asks whether standard benchmarks can see the architectural differences that matter. To find out, it reports 64 from-scratch pretraining runs of Transformers and three linear-model families, at 1.3-8B parameters on 1-2T Nemotron-CC tokens. The cost was roughly 100k USD per 8B run.
The first finding is about scale. In academic-scale pretraining, meaning 1.3B parameters and 100B tokens, architectural differences can all but disappear when the models are carefully tuned. At 1-2T tokens they become measurable. Canon layers, which add local token mixing, consistently improve Transformers. At 8B, they save about 30% of pretraining tokens for the same MMLU accuracy.
The second finding concerns linear models. The authors introduce GLA5 and GDN2, which improve gated linear attention and Gated DeltaNet with low-rank output gates. Surprisingly, once Canon layers are added, the simplest linear model, gated linear attention, matches Gated DeltaNet and outperforms Mamba2, even at 8B parameters and 1T tokens. From this the authors infer that the extra machinery of Gated DeltaNet and Mamba2 may do little more than mix neighboring tokens.
The third finding is a warning about benchmarks. Even at 8B/1T, high scores on standard benchmarks can hide weaknesses in basic in-context skills. Linear models that match or beat Transformers on those benchmarks cannot reliably retrieve information from a short 500-token context. And no model, Transformer or linear, reliably beats random guessing on what the authors call arguably the simplest two-hop question, stated in a few short sentences, even at this scale.
The authors conclude that these results challenge traditional architecture research based primarily on standard benchmark scores. Small synthetic playgrounds, which predicted the architectural differences above at under 100M parameters, can reveal what the benchmarks miss. For practitioners, one lesson is to include in-context reasoning examples in pretraining data from an early stage, rather than reserve them for later mid-training, let alone post-training.
Key facts
- The study covers 64 from-scratch pretraining runs of Transformers and three linear-model families at 1.3-8B parameters on 1-2T Nemotron-CC tokens, at roughly 100k USD per 8B run.
- Canon layers, which add local token mixing, consistently improve Transformers and save about 30% of pretraining tokens for the same MMLU accuracy at 8B.
- With Canon, gated linear attention matches Gated DeltaNet and outperforms Mamba2 even at 8B/1T; the authors also introduce GLA5 and GDN2, which add low-rank output gates.
- Linear models that match or beat Transformers on standard benchmarks cannot reliably retrieve information from a short 500-token context, and no model reliably beats random guessing on a simple two-hop question.
- Small synthetic playgrounds predicted the architectural differences at under 100M parameters; the authors advise including in-context reasoning examples in pretraining data early.
Why it matters
Architecture comparisons are often run at academic scale, and the authors report that at 1.3B parameters and 100B tokens the differences can all but disappear when models are carefully tuned. At 1-2T tokens they become measurable, which suggests small-budget results may not tell the whole story. The paper also questions how much of the extra machinery in Gated DeltaNet and Mamba2 is needed: with Canon layers, plain gated linear attention matches the first and beats the second at 8B/1T. And it argues that standard benchmark scores can hide basic failures, which challenges architecture research built mainly on those scores.
Who it affects
Teams that pretrain language models and choose between Transformers and linear-attention or state-space designs are the main audience. So are researchers who judge architectures by benchmark scores at small scale. Anyone who runs expensive pretraining will care about the reported token saving: about 30% for Transformers with Canon layers at 8B, measured on MMLU.
How to use it
The authors' stated lesson for practitioners is to include in-context reasoning examples in pretraining data from an early stage, rather than reserve them for later mid-training or post-training. They also point to small synthetic playgrounds, which predicted the architectural differences at under 100M parameters, as a cheaper way to reveal what standard benchmarks miss. For Transformers, Canon layers are the intervention to look at. The source text does not mention a code, model or dataset release.
How solid is it
The evidence is a large controlled study: 64 from-scratch runs across Transformers and three linear-model families, up to 8B parameters and 1-2T tokens, at roughly 100k USD per 8B run. That is an unusually costly setup for an open comparison. The account here rests on the paper's abstract, so the detailed numbers behind the claims are not visible. The 30% saving is stated for Transformers with Canon at 8B on MMLU only. The inference that Gated DeltaNet and Mamba2 add little beyond neighbor mixing is hedged by the authors themselves with "may". It is a preprint, and the abstract gives no benchmark scores for any model.
Risks and caveats
The token saving is reported for one benchmark (MMLU) at one size (8B), and the source gives no figure for other sizes or benchmarks. The findings are for models up to 8B parameters and 2T tokens, so the source does not support claims beyond that range. The two-hop result says only that no model reliably beats random guessing; the source gives no scores. The authors' view that synthetic playgrounds can reveal what benchmarks miss is their own claim, as is the advice on early in-context reasoning data. The abstract also does not say what the 100k USD per run covers.
“high scores on standard benchmarks can hide weaknesses in basic in-context skills”
— Paper abstract, Physics of Language Models: Part 4.2