Glonce

Paper proposes attention-free Causal Grassmann layer, within about 10-15% of Transformers on Wikitext-2

researchThe Glonce editorsHacker News🔥64Glonce rating64Impact45Novelty60Relevance95Surprise50Players15Credibility75

A paper submitted to arXiv (cs.LG) on 22 Dec 2025, titled "Attention Is Not What You Need", asks a basic question about sequence modeling: is explicit self-attention actually necessary for strong performance and reasoning?

The paper's argument starts with a reading of attention. It holds that standard multi-head attention is best seen as a form of tensor lifting: hidden vectors are mapped into a high-dimensional space of pairwise interactions, and learning proceeds by constraining that lifted tensor through gradient descent. That mechanism, the paper says, is extremely expressive but mathematically opaque, because after many layers it becomes very hard to describe the model with a small family of explicit invariants.

As an alternative, the paper proposes an attention-free architecture built on Grassmann flows. Instead of forming an L by L attention matrix, its Causal Grassmann layer does three things. First, it linearly reduces token states. Second, it encodes local token pairs as two-dimensional subspaces on a Grassmann manifold, using Plucker coordinates. Third, it fuses these geometric features back into the hidden states through gated mixing. Information then propagates by controlled deformations of low-rank subspaces over multi-scale local windows, so the core computation lives on a finite-dimensional manifold rather than in an unstructured tensor space.

Two experiments are reported. On the Wikitext-2 language modeling benchmark, purely Grassmann-based models with 13 to 18 million parameters reach validation perplexities within about 10 to 15 percent of size-matched Transformers. On the SNLI natural language inference task, a Grassmann-Plucker head on top of DistilBERT slightly outperforms a Transformer head: best validation and test accuracies are 0.8550 and 0.8538, against 0.8545 and 0.8511 for the Transformer head.

The paper also analyzes the complexity of Grassmann mixing and shows linear scaling in sequence length for fixed rank. Its closing argument is that manifold-based designs offer a more structured route toward geometric and invariant-based interpretations of neural reasoning.

Key facts

  • The paper (arXiv cs.LG, submitted 22 Dec 2025) proposes a Causal Grassmann layer that replaces the L by L attention matrix with Plucker-coordinate features of local token pairs on a Grassmann manifold.
  • On Wikitext-2, purely Grassmann-based models with 13 to 18 million parameters reach validation perplexity within about 10 to 15 percent of size-matched Transformers.
  • On SNLI, a Grassmann-Plucker head on DistilBERT scores 0.8550 validation and 0.8538 test accuracy, versus 0.8545 and 0.8511 for a Transformer head.
  • The authors show linear scaling in sequence length for fixed rank and argue the design offers a more structured route to geometric, invariant-based interpretations of neural reasoning.

Why it matters

Self-attention is the core of nearly every modern language model, so a credible attention-free design is worth a look. The paper's pitch is less about beating Transformers than about structure: it argues attention is mathematically opaque after many layers, while computation on a finite-dimensional manifold could be easier to describe with explicit invariants. It also reports linear scaling in sequence length for fixed rank, in contrast to the L by L attention matrix it sets out to avoid.

Who it affects

Mainly researchers working on sequence architectures, efficient alternatives to attention, and interpretability, especially those drawn to geometric or invariant-based accounts of how networks reason. Practitioners who build on Transformers have nothing to change on the strength of this paper alone.

How to use it

There is nothing to deploy yet. The paper is available on arXiv as a PDF and an experimental HTML version, and it is useful as a reference for the Causal Grassmann layer's construction: linear reduction of token states, Plucker-coordinate encoding of local token pairs, and gated mixing back into hidden states. No code release is mentioned in the source.

How solid is it

It is a preprint, and no peer review, code release or independent replication is mentioned. The Wikitext-2 result is a gap, not a win: Grassmann models are within about 10 to 15 percent of Transformers, which means they do not match or beat them, and the abstract gives no absolute perplexity values. The SNLI edge is tiny: 0.8550 versus 0.8545 on validation (0.05 percentage points) and 0.8538 versus 0.8511 on test (0.27 points), with no significance testing or multiple runs mentioned. The abstract names no authors or institutions.

Risks and caveats

The tested models are small, 13 to 18 million parameters, and no results at larger scale are given. The SNLI model puts the Grassmann head on top of DistilBERT, which is itself a Transformer, and the abstract does not say that this model is attention-free. No wall-clock speed, memory or hardware figures are given, so linear scaling in sequence length for fixed rank is a complexity claim rather than a measured speedup. The interpretability benefit is something the authors argue for, not something the abstract demonstrates.

“This mechanism is extremely expressive but mathematically opaque, because after many layers it becomes very hard to describe the model with a small family of explicit invariants.”

— From the paper's abstract, on standard multi-head attention