Directional decomposition reveals space-dependent asymmetry in transformer edits

Transformer representations change layer by layer through additive updates: attention and MLP blocks each add a vector to the running hidden state, and the attention mechanism itself aggregates value vectors from other tokens. A paper posted to Hugging Face treats this evolution as a functional geometry problem and decomposes every one of these learned updates into two parts: a component parallel to the representation's current direction, which reinforces it, and a component perpendicular to it, which redirects it.

The decomposition is applied in two different spaces. The first is the standard one: attention and MLP updates measured relative to the hidden state they are added to. The second is specific to attention itself: the aggregation of value vectors measured relative to the current token's own value. Across pretrained models, the authors report substantial parallel components beyond the residual identity path, meaning a meaningful share of what each layer adds to the residual stream reinforces the direction the representation already had, rather than only perturbing it sideways.

The practical payoff shows up in targeted edits to these components. Robustness depends sharply on where the edit lands: adjusting the parallel component in value space while excluding a token's own value from the adjustment, which the paper calls exclude-self, is markedly more robust than adjusting the parallel component in residual space or adjusting either perpendicular component. The exclude-self, value-space approach leaves a token's direct self-message untouched and scales only the aggregate contribution coming from other tokens, which the authors present as the reason it holds up better under editing.

The same decomposition doubles as a diagnostic for compression. Applied to the error that a compression method introduces into an update, its perpendicular component separates different compression methods from one another more clearly than the parallel component does, making perpendicular error the more informative signal for judging which compression approach damages a model's representations more.

Finally, the authors test the decomposition as a training-time intervention: suppressing the full-aggregate parallel component while pretraining a model from scratch lowers its validation-loss trajectory and improves its downstream task averages, with the value-space version of the suppression producing the strongest effect among the variants tried. The abstract does not name which pretrained models were studied, and it does not specify the dataset, model size, or training budget behind the from-scratch pretraining experiment; its findings are reported in qualitative terms such as substantial, markedly more robust, and strongest, rather than with percentages or benchmark scores. Code for the decomposition is available in the project's public GitHub repository, at github.com/Shwai-He/Transformer-Geometry.

Key facts

The paper decomposes every additive update inside a transformer, in attention and MLP layers and in attention's value aggregation, into a component parallel to the representation's current direction and one perpendicular to it.
Across pretrained models, the authors find substantial parallel components beyond the residual identity path, meaning much of what each layer adds reinforces the existing direction rather than only perturbing it.
Editing the parallel component of value-space aggregation while excluding a token's own value, called exclude-self, is markedly more robust than editing in residual space or along either perpendicular direction.
For compression, perpendicular error separates different compression methods more clearly than parallel error does, making it the more useful diagnostic signal.
Suppressing the full-aggregate parallel component during from-scratch pretraining lowers validation loss and improves downstream task averages, with the value-space variant producing the strongest effect.

Why it matters

Work on editing or compressing transformer representations usually treats the residual stream as something largely preserved, with edits or compression error read as small perturbations layered on top of it. This paper's central finding cuts against that framing: a substantial share of what each layer adds is already parallel reinforcement of the existing direction, not passive pass-through. Treating updates as a parallel-versus-perpendicular split turns editing, compression and training into instances of one underlying geometric question, how much a given change alters direction rather than magnitude, and the same decomposition then explains three separately useful things: which edits survive, which compression errors are diagnostic, and which training-time suppression improves outcomes.

Who it affects

Researchers and engineers doing model editing or activation steering gain a specific, markedly more robust technique: exclude-self manipulation of the parallel component in value space. Teams building or evaluating compression methods, such as quantization, pruning or distillation, gain a new way to compare methods by their perpendicular error rather than a single aggregate error figure. Anyone pretraining a language model from scratch has a candidate intervention, parallel suppression in value space, reported to improve both validation loss and downstream averages.

How to use it

The paper is posted on Hugging Face Papers, and the authors have released code for the decomposition in a public GitHub repository, at github.com/Shwai-He/Transformer-Geometry. Applying the parallel and perpendicular split to a different model's own updates means working from that repository and the paper's method description directly: the abstract summarizes the results, not a step-by-step recipe.

How solid is it

The claims come only from the paper's own abstract, and every one of them is qualitative: substantial, markedly more robust and strongest appear with no accompanying percentages, loss values or benchmark scores, so the size of any effect cannot be checked from the abstract alone. The abstract does not name which pretrained models were studied, and it does not give the dataset, model size or training budget behind the from-scratch pretraining result, which limits how far the reported improvements can be assumed to generalize. The listing has drawn minimal outside engagement so far, a handful of points and a single comment, so it has not yet had independent scrutiny beyond the authors' own description.

Risks and caveats

Because the abstract does not name the paper's authors or institutions, the specific pretrained models used, or any numeric result, the work cannot currently be checked against a benchmark table or reproduced from the public description alone; anyone who needs actual magnitudes has to consult the full paper or the linked code rather than this abstract. The from-scratch pretraining intervention is reported at an unspecified scale, so it remains unclear whether the validation-loss and downstream-average gains hold at the model sizes used in production systems. As with any single paper's self-reported results, the markedly more robust and strongest characterizations are the authors' own framing, not yet corroborated independently here.

“exclude-self value-space parallel manipulation is markedly more robust than residual-space and perpendicular counterparts, preserving the direct self message while scaling only the non-self aggregate.”

— the paper's abstract