Glonce

Paper tracks layer-by-layer paths in Qwen, Llama and Gemma inference

researchThe Glonce editorsarXiv🔥57Glonce rating57Impact42Novelty62Relevance96Surprise28Players10Credibility70

Mechanistic studies of large language models usually localize function in parameters, attention heads or neurons. The authors of this arXiv paper argue that those parts do not specify how a single inference unfolds. Instead, they treat the ordered sequence of layer states as the empirical process and test whether layer order defines a measurable computation.

The first result concerns alignment. Across representative Qwen, Llama and Gemma checkpoints, the realized layer order produced paths that were 1.60 to 2.24 times as aligned with their endpoint chord as endpoint-preserving shuffles. The order effect also transferred to a lexical task. It was stronger, though, in three random initializations of the same Qwen architecture. The authors read that as an architecture-compatible scaffold rather than a law that comes from training.

On the choice of coordinates, direct hidden states gave stronger mechanism coordinates than vocabulary readbacks, and a task-defined coordinate made candidate competition computable.

The intervention part came next. Additive interventions failed. Local transition actuators then selectively controlled internal trajectories across checkpoints. A confidence gate drove 17 of 87 held-out Qwen closure trajectories across the full-vocabulary top-1 candidate boundary, and it left all 203 non-closure top-1 outputs unchanged. Safety-matched action reassignment performed similarly, which the authors take to place the gain in selective gating and the bounded actuator family, not in fine-grained dose assignment.

The authors state the limits of the claim themselves: layer order exposes an observable, computable and behaviourally consequential process of transformer inference, without establishing a complete theory of LLM dynamics or control of answer correctness.

Key facts

  • Across representative Qwen, Llama and Gemma checkpoints, realized layer order gave paths 1.60 to 2.24 times as aligned with their endpoint chord as endpoint-preserving shuffles.
  • The order effect transferred to a lexical task and was stronger in three random initializations of the same Qwen architecture, which the authors read as an architecture-compatible scaffold, not a training-specific law.
  • Additive interventions failed; local transition actuators then selectively controlled internal trajectories across checkpoints.
  • A confidence gate moved 17 of 87 held-out Qwen closure trajectories across the top-1 candidate boundary and left all 203 non-closure top-1 outputs unchanged.
  • The authors say the work does not establish a complete theory of LLM dynamics or control of answer correctness.

Why it matters

Most interpretability work asks where a function lives: in a parameter, a head or a neuron. This paper asks a different question, namely how one inference unfolds from layer to layer, and it measures the answer. The headline finding is that the real layer order matters: paths follow their endpoint chord 1.60 to 2.24 times as closely as endpoint-preserving shuffles do. The stronger effect in randomly initialized models of the same Qwen architecture points to the architecture, not training, as the source of much of the structure.

Who it affects

The work is aimed at researchers in mechanistic interpretability and those who study how transformers compute internally. It spans Qwen, Llama and Gemma checkpoints, so the framing is not tied to one model family. The confidence-gate result is reported for Qwen only.

How to use it

The source gives no practical recipe, code or deployment path. What a reader can take from it is the method: treat the layer-state trajectory of one inference as the object of study, compare real layer order against endpoint-preserving shuffles, and use direct hidden states rather than vocabulary readbacks as coordinates. The paper reports that additive interventions failed while local transition actuators worked.

How solid is it

This is an arXiv preprint, and the account here rests on its abstract. The headline measurement is a range of 1.60 to 2.24 times across checkpoints; per-checkpoint values are not given. The intervention result is 17 of 87 held-out Qwen closure trajectories, with all 203 non-closure top-1 outputs unchanged. The claims are the authors' own, and they limit them explicitly.

Risks and caveats

The authors say the work does not establish a complete theory of LLM dynamics or control of answer correctness, so the 17 shifted trajectories should not be read as 17 corrected answers. The confidence-gate result covers Qwen, and 17 of 87 is a minority of the closure trajectories tested. The abstract does not define 'closure' trajectories or the lexical task, and it gives no model sizes or exact checkpoint names.