Glonce

LLMs commit to relation type before entity in factual recall

researchThe Glonce editorsarXiv🔥71Glonce rating71Impact60Novelty70Relevance95Surprise65Players20Credibility80

A new study asks whether relation-type information (the kind of fact being recalled, such as capital-of) and entity-specific information (the actual answer, such as France pointing to Paris) become active at the same point inside a language model, or at different points. Using four complementary causal diagnostics, tested across four decoder-only models and eight prompt families, the authors find a consistent temporal split: relation information becomes generation-controlling before entity information does. At a detection threshold of 0.4, relation onset precedes entity onset by 10 to 16 layers, a gap equal to 31 to 44% of the network's total depth. This ordering is not a one-off result: it holds across all 16 model-threshold combinations the authors tested, spanning thresholds from 0.2 to 0.5.

The entity information is not simply missing in the early layers where it lags behind. Patching the correct entity token into the model's computation succeeds 90 to 100% of the time even in those early layers, showing the information is present and recoverable. What changes later is whether that information actually drives what the model outputs at the final token. The authors describe this as deferred commitment: entity information sits available at the entity's own token position from early on, and only becomes generation-controlling at the final token once it has been routed there.

Key facts

  • Across four decoder-only models and eight prompt families, tested with four causal diagnostics, relation-type information becomes generation-controlling 10 to 16 layers before entity-specific information does, at threshold 0.4.
  • That gap corresponds to 31 to 44% of the network's total depth.
  • The relation-before-entity ordering held across all 16 model-threshold combinations tested, for thresholds ranging from 0.2 to 0.5.
  • Entity information is not absent in early layers: patching the correct entity token there succeeds 90 to 100% of the time, but it does not yet control the final output.
  • The authors call this deferred commitment: entity information is available at the entity's own token position but only becomes generation-controlling at the final token after being routed there.

Why it matters

Interpretability work on factual recall often treats it as a single retrieval step: the model looks up a fact and outputs it. This paper's causal tracing splits that step into two temporally distinct stages. First, the model settles on what kind of relation is being asked about; only several layers later does the specific entity that answers it start controlling generation. Because the ordering holds across four different models, eight prompt families and a range of detection thresholds, it reads as evidence for a general two-stage recall pipeline inside these networks rather than a quirk of one model or one way of measuring it.

Who it affects

The direct audience is researchers doing mechanistic interpretability and causal tracing on language models, along with people building knowledge-editing methods that try to correct or update a specific fact a model has memorized. Knowing that relation-type and entity information become controlling at different depths, and by how many layers, matters for deciding which layers an editing method should target and what it is actually changing when it does.

How to use it

The paper's contribution is a measurement protocol rather than a tool: four causal diagnostics applied at the entity-token position and at the final token, swept across detection thresholds from 0.2 to 0.5. Researchers working on similar questions can reuse this setup, comparing the layer at which entity-token patching first succeeds against the layer at which it starts controlling the final-token output, to check whether the same staged pattern shows up in other models or prompt types.

How solid is it

The core finding replicates across four decoder-only models, eight prompt families and four separate causal diagnostics, and the relation-before-entity ordering holds in all 16 model-threshold combinations tested, across thresholds 0.2 to 0.5. That is a fairly wide sweep for a single paper. The text made available here is the abstract, so it does not describe the diagnostics, the models or the prompt families in enough detail to independently judge experimental design choices beyond what is stated.

Risks and caveats

The available text does not name the four models, the four diagnostics or the eight prompt families, does not give a publication or submission date, and lists no author names or institutional affiliations. It also does not explain the underlying mechanism by which entity information gets routed from the entity-token position to the final token, only that the routing appears to happen later than relation commitment.

“Entity commitment to generation is deferred: entity information is available at the entity-token position but becomes generation-controlling at the final token only after being routed there.”

— the paper's abstract