Study finds transformer LLMs can output a superposition of two predictions at once
A new paper argues that large language models, despite being built from highly non-linear components like attention and nonlinear activations, exhibit a surprising linear behavior at the level of their output predictions. When inputs from two distinct text streams are linearly combined before being fed into the model, the resulting output is a superposition of the individual next-token distributions each stream would have produced on its own. The authors term this the Superposition Linearity Hypothesis.
The paper presents this as an intrinsic property of the Transformer architecture itself rather than something that emerges only after training on large corpora. In fact, the authors report the opposite trend: the superposition effect tends to diminish as pretraining progresses, meaning models trained longer on standard objectives become less linear in this respect.
However, the authors show that this linearity can be substantially restored through lightweight fine-tuning, which significantly reduces the divergence between what the model actually predicts and the average of the two individual next-token distributions it would have produced for each stream separately.
Building on this, the paper introduces a guided decoding procedure that disentangles the superposed output, allowing the model to simultaneously generate two coherent, separate continuations from a single forward pass, effectively reading two predictions out of one pass through the network instead of running the model twice.
The available material does not name the authors, their institutions, a publication or submission date, the models or model sizes tested, or any quantitative results for the divergence reduction or the decoding procedure; it describes the findings only in qualitative terms.
Key facts
- Linearly combining inputs from two distinct text streams causes an LLM's output to become a superposition of the individual next-token distributions, a phenomenon the authors call the Superposition Linearity Hypothesis.
- The authors present this as intrinsic to the Transformer architecture rather than a byproduct of training.
- The effect diminishes as pretraining progresses, but lightweight fine-tuning substantially restores it, significantly reducing divergence from the average of the two individual next-token distributions.
- A newly introduced guided decoding procedure disentangles the superposed output, letting the model generate two coherent continuations from a single forward pass.
- No author names, affiliations, publication date, model names or sizes, or quantitative metrics are given in the available material.
Why it matters
The paper's core claim is that language models, though built from non-linear parts, behave in a linear way at the output level when their inputs are linearly mixed: the prediction splits cleanly into a superposition of what each input stream would have predicted alone. That is a mechanistic finding about how Transformers represent and process information, and it ties into the broader interpretability question of what internal structure makes a model's outputs decomposable this way.
Who it affects
The finding is aimed at researchers working on mechanistic interpretability and model internals, and at engineers designing decoding or inference pipelines who might exploit a single forward pass to extract more than one prediction.
How to use it
The paper's practical payoff is a guided decoding procedure that disentangles the superposed output so the model produces two coherent, independent continuations from one forward pass rather than two separate passes. The available text describes this only by its stated function; no details of how the procedure works, which models it was tested on, or how much compute it saves are given.
How solid is it
The account rests on the paper's own abstract-level description of its findings; it states its claims as demonstrated results but the available material carries no author names, institutional affiliations, publication date, or supporting numbers, so the specifics cannot be independently checked from what is here.
Risks and caveats
No quantitative results are given for how much the fine-tuning reduces divergence, nor for how well the guided decoding procedure performs, and no model names or sizes tested are specified, so the generality of the effect across model families and scales is unclear from the available material.
“when inputs from distinct text streams are linearly combined, the model outputs a superposition of the individual next-token distributions”
— the paper