Latent Lattice Language Models lower perplexity at 406M parameters
A project page with affiliations listed as MIT and Adobe introduces Latent Lattice Language Models (LLLMs). The idea is to raise a model's computational capacity, meaning the number of active computational states it keeps, instead of the usual routes of deeper or looped layers, latent reasoning steps or richer residual connections.
The mechanism is called the Lattice of Latents: a set of per-layer horizontal latent streams that run recurrently along time, next to the vertical residual stream that flows through depth. The authors compare it with two families of prior work. Like hyper-connections, mHC, and attention or value residuals, the Lattice enhances the residual stream. But rather than adding or rerouting residual pathways through depth, it provides companion computational states that persist across tokens. And unlike hybrid Transformer-SSM architectures, whose recurrent states mainly serve as internal memory for sequence mixing, these latents directly condition the vertical stream's attention, KV-cache writes, MLP computation and decoding.
The headline result: on Transformers trained on 20B FineWeb-Edu tokens, a 406M-parameter Lattice model lowers validation perplexity from 15.41 to 14.40 over a parameter-matched Transformer. The gains are said to extend across architectural variants. The Lattice also composes with earlier residual-stream enhancements: with hyper-connections and input residuals, validation perplexity falls from 15.41 (base Transformer) and 14.22 (Transformer with the same enhancements) to 13.61.
On recurrent models, the page tests Gated DeltaNet (GDN), Mamba-2, Triadic GDN and GDN-Mamba-2 hybrids at about 405M parameters and 10B FineWeb-Edu tokens. Each Lattice model is iso-parameter with its pure counterpart. The authors say the Lattice improves perplexity at every tested native state size, and that the Lattice-GDN and Lattice-Triadic-GDN variants outperform pure baselines with 4x larger native states. Native-state size here counts recurrent-memory entries per mixer (1K = 1024 scalars) and excludes the horizontal latents and convolution states.
Two further comparisons appear in the interactive figures. In the 400M-model, 20B-token setup, the Lattice models narrow the vertical width from 1024 to 932 to stay iso-parameter. Both LLLMs use 22 layers against 40 for the 40-layer Transformer, so at the same attention width (1024) they keep a 45% smaller KV cache. In a harder regime the vertical residual stream is squeezed to 256 dimensions, with 10B-token pretraining; the page's figure adds the horizontal-update enhancements one at a time, and the caption says they recover most of the gap to a full-width Transformer.
The GFLOP/token figures on the page exclude attention scores, which add about 0.1 GFLOP/token, and the embedding lookup. The authors conclude that persistent per-layer state is a complementary axis for scaling language-model computation.
Key facts
- A 406M-parameter Lattice model reaches validation perplexity of 14.40 versus 15.41 for a parameter-matched Transformer, both trained on 20B FineWeb-Edu tokens.
- Combined with hyper-connections and input residuals, the Lattice reaches 13.61, against 15.41 for the base Transformer and 14.22 for a Transformer with the same enhancements.
- On Gated DeltaNet, Mamba-2, Triadic GDN and GDN-Mamba-2 hybrids (about 405M parameters, 10B tokens), the Lattice improves perplexity at every tested native state size; two GDN variants beat pure baselines with 4x larger native states.
- The two LLLMs use 22 layers versus 40 for the Transformer and keep a 45% smaller KV cache at the same attention width of 1024.
- The core change: per-layer horizontal latent streams persist across tokens and directly condition attention, KV-cache writes, MLP computation and decoding.
Why it matters
Most routes to more computation per token go through depth, looping or reasoning steps. This work argues for a different axis: keep more active computational states alive across tokens, one set per layer. If the reported gains hold, persistent per-layer state would sit alongside depth and residual-stream design as a lever for language-model quality. The authors also set it apart from hybrid Transformer-SSM models, whose recurrent states mainly act as internal memory for sequence mixing, by letting the latents shape attention, KV-cache writes, MLP computation and decoding.
Who it affects
Mainly researchers working on language-model architecture: residual-stream variants such as hyper-connections, mHC and value residuals, and recurrent or state-space mixers such as Gated DeltaNet and Mamba-2. The KV-cache result (45% smaller at the same attention width, with 22 layers instead of 40) is also relevant to anyone weighing inference memory cost, though the page reports it for a single comparison.
How to use it
For now this is something to read and probe rather than deploy. The page offers interactive charts where you can switch between training and inference, compare models and hover over bars for details. No code, model release or license is mentioned in the text. A practitioner could take away the design pattern: a recurrent latent stream per layer beside the residual stream, injected into attention and MLP, and tested against parameter-matched baselines.
How solid is it
The numbers are specific and consistent with a controlled comparison: parameter-matched or iso-parameter baselines, the same FineWeb-Edu training data, and results across several architectures. But this is a project page, not an independently checked result. No author names are given; only the affiliations MIT and Adobe appear. No publication or submission date, venue, or peer-review status is stated. Only validation perplexity is reported, and the text does not say how the Lattice performs on downstream tasks. No independent replication or outside reaction is mentioned.
Risks and caveats
The scale is small: results are around 400M parameters and 10B to 20B tokens, and no results at larger scales are reported. Perplexity gains of about one point (15.41 to 14.40) may not carry over to larger models or to downstream tasks. The Lattice models narrow the vertical width (1024 to 932) to stay iso-parameter, and no wall-clock training or inference speed figures are given in the text, so the practical cost of the recurrent streams is not shown there. The 45% KV-cache saving comes from using fewer layers, and the page compares it to one specific 40-layer Transformer.
“These results identify persistent per-layer state as a complementary axis for scaling language-model computation.”
— Abstract, Latent Lattice Language Models project page