Glonce

KAIST study finds AI reasoning steps leave distinct patterns in model layers

researchThe Glonce editorsThe Decoder🔥70Glonce rating70Impact60Novelty72Relevance98Surprise45Players50Credibility78
KAIST study finds AI reasoning steps leave distinct patterns in model layers

Researchers at South Korea's KAIST and Naver AI Lab tested whether the distinct reasoning steps a language model writes out in text, such as reading data, breaking down a problem, retrieving a formula, or running a calculation, also show up as separable patterns inside the model's internal numerical representations. They found that they do, and the separation is strongest in the middle layers.

The team defined eight recurring reasoning operations, among them extraction, decomposition, formula recall, deduction and computation. They had three models, Qwen2.5-7B, Qwen3-8B and Gemma4-31B, solve math problems, split each solution into segments, then used GPT-5 to label every segment with one of the eight operations. A classifier trained on the models' internal representations could reliably tell the operations apart across all three models, and it outperformed a classifier that only looked at the words used. The step's position within the solution path did not explain the separation either, meaning the internal states hold information about the type of reasoning step that goes beyond surface wording.

Common function words such as 'a,' 'is' or 'the' appear across very different reasoning steps. In early layers their internal representations are jumbled together, but by the middle and later layers the same word is represented differently depending on which reasoning step it belongs to. The researchers also blocked attention to the preceding 30 tokens through a targeted intervention; the signal for the current reasoning step weakened, showing that a step does not form in isolation but builds on the context before it. Even on problems the models solved incorrectly, the type of step stayed identifiable: a flawed computation step still looked like a computation step internally, even though the result was wrong.

The separability finding also replicated on Llama-3-8B, and for Qwen3-8B the classifiers trained on math problems transferred successfully to the GPQA-Diamond and MATH-500 benchmarks. The article notes that the experiments cover only math tasks and a handful of models, and that whether the finding can be used to catch errors or steer a model mid-generation remains an open question for future work rather than a demonstrated capability.

The result speaks to AI safety debates about chain-of-thought oversight. OpenAI has pointed to reading a model's written reasoning as one of the few oversight tools available, but Anthropic has shown that models disclose the hints they actually used in only 25 to 39 percent of cases. Separately, a method that translates a model's internal vectors into readable text found that Claude Opus 4.6 processes more than what shows up in its output reasoning, and OpenAI's Astra model uses a technique called Recurrent Depth that shifts part of its reasoning into internal numerical representations, the same space the KAIST study examines.

Key facts

  • KAIST and Naver AI Lab researchers found that reasoning steps a model writes out in text, among them extraction, decomposition, formula recall, deduction and computation (eight operations in total), also form separable patterns inside its internal representations, strongest in the middle layers.
  • The team tested three models, Qwen2.5-7B, Qwen3-8B and Gemma4-31B, on math problems, using GPT-5 to label each solution segment by operation; a classifier reading internal representations outperformed one that only looked at the words used, and the step's position in the solution did not explain the separation either.
  • Blocking attention to the preceding 30 tokens through a targeted intervention weakened the signal for the current reasoning step, showing that steps build on prior context rather than forming in isolation.
  • The type of reasoning step stayed identifiable internally even on incorrectly solved problems, and the finding replicated on a fourth model, Llama-3-8B; for Qwen3-8B the trained classifiers also transferred to the GPQA-Diamond and MATH-500 benchmarks.
  • The article notes the experiments cover only math tasks and a handful of models and calls using the signal to catch errors or steer generation an open question; the topic matters for AI safety because Anthropic has found models disclose the reasoning hints they actually used in only 25 to 39 percent of cases.

Why it matters

Chain-of-thought text is one of the few windows into how a reasoning model reaches an answer, and OpenAI has pointed to reading that text as one of the few oversight tools available. Anthropic has already shown that models disclose the hints they actually used in only 25 to 39 percent of cases, and a separate method that reads a model's internal vectors found that Claude Opus 4.6 processes more than what shows up in its written reasoning. The KAIST and Naver AI Lab study adds another piece: it shows that the distinct steps a model narrates in text, such as extraction, decomposition or computation, leave correspondingly distinct traces inside the model's numerical representations, most clearly in the middle layers. Linking what a model says it is doing to what its internal state shows it is doing is a precondition for any future tool that checks a model's reasoning against its internals rather than trusting the text alone.

Who it affects

The direct audience is interpretability and AI safety researchers working on chain-of-thought monitoring, along with labs such as OpenAI and Anthropic whose oversight arguments the finding bears on. It changes nothing for people who use models like Qwen, Gemma or Llama day to day: none of the four tested models, Qwen2.5-7B, Qwen3-8B, Gemma4-31B and Llama-3-8B, gained a new capability. The result is a measurement of how these models work internally, not a product change.

How to use it

There is nothing to install or apply yet. The researchers built classifiers that read a model's internal representations well enough to tell reasoning operations apart, and for Qwen3-8B those classifiers also transferred successfully to two further benchmarks, GPQA-Diamond and MATH-500. That transfer is the closest thing to a usable technique here, and even it is a research result rather than a released tool: the article states plainly that using this kind of signal to catch reasoning errors or steer a model mid-generation remains an open question for future work.

How solid is it

The core finding held across three separately trained models, Qwen2.5-7B, Qwen3-8B and Gemma4-31B, then replicated on a fourth, Llama-3-8B. Before crediting internal representations, the researchers ruled out two simpler explanations: a classifier using only the words in each step performed worse than one using internal representations, and the step's position within the solution path did not account for the separation either. They also stress-tested the finding by blocking attention to the preceding 30 tokens, which weakened the signal for the current step, and by checking incorrectly solved problems, where a flawed computation step still read internally as a computation step even though the answer was wrong. For Qwen3-8B, the trained classifiers additionally transferred to the GPQA-Diamond and MATH-500 benchmarks.

Risks and caveats

The article itself limits the claim: the experiments cover only math tasks and a handful of models, and whether the underlying signal can be used to catch reasoning errors or steer a model mid-generation is an open question, not something already demonstrated. It also does not quantify how well the classifiers distinguish reasoning steps: descriptions like 'reliably told apart' and 'performed worse' come with no accuracy number attached. No individual researchers, paper title, preprint venue or peer-review status are named, only the institutions KAIST and Naver AI Lab, and no explanation is given for why the separability specifically peaks in the middle layers.