Glonce

Study finds RL fine-tuning gives math reasoning models deeper, more structured representations than SFT

researchThe Glonce editorsarXiv🔥69Glonce rating69Impact60Novelty70Relevance100Surprise45Players25Credibility85

Reasoning models trained with reinforcement learning (RL) have increasingly been shown to outperform supervised fine-tuned (SFT) models on mathematical reasoning tasks, but the paper's authors note the mechanistic reason for that gap has stayed unclear. To find it, they ran two lines of analysis on the models' internal representations. First, they trained linear probes on the layer-wise hidden states of both model types and used them to predict whether a given answer was correct. The RL models' probes achieved higher accuracy at this than the SFT models' probes, which the authors read as evidence that RL models build representations that are more linearly separable and more structured. Second, they ran mean ablation studies, replacing activations with their mean value to see which layers matter most. RL models showed a hierarchical pattern in which deeper layers become progressively more critical to the output, while SFT models spread importance roughly evenly across all layers. Taken together, the authors argue RL training does not just improve accuracy incrementally; it fundamentally restructures how a model represents and processes a reasoning problem internally. The paper adds a third analysis on top of this: token-count variability under repeated sampling of the same problem, used as a proxy for adaptive compute allocation. Here the results were mixed rather than uniform. Some RL-tuned models showed much higher variability in how many tokens they spent per attempt than their SFT counterparts, but other RL-tuned models were just as consistent as SFT models. The authors conclude that this variability tracks the overall training pipeline more than the RL-versus-SFT split by itself, and they read the pattern as exposing which models have settled on a stable reasoning policy versus which remain under-determined, with solution behavior that may not be fully identifiable from the training method alone.

Key facts

  • Linear probes trained on layer-wise hidden states predict answer correctness more accurately for RL-tuned models than for SFT-tuned models, indicating more structured, linearly separable representations in the RL models.
  • Mean ablation studies show RL models develop a hierarchical structure where deeper layers grow progressively more critical, while SFT models distribute importance evenly across layers.
  • The authors conclude RL training fundamentally restructures how a model represents and processes reasoning problems, rather than just improving accuracy at the margins.
  • A separate analysis of token-count variability under repeated sampling finds higher variability in some RL-tuned models than their SFT counterparts, but strong consistency in others.
  • The authors attribute this token-allocation variability more to the overall training pipeline than to RL versus SFT alone, treating it as a signal for which models have a stable reasoning policy versus an under-determined one.

Why it matters

RL-tuned reasoning models have been outperforming SFT models on math benchmarks for a while, but nobody had pinned down why at the representational level. This paper gives a concrete, testable answer: RL training pushes models toward more linearly separable, hierarchically organized internal representations, with deeper layers carrying more of the decision-relevant signal. That reframes the RL-versus-SFT gap as a difference in internal structure, not just a difference in output accuracy.

Who it affects

The findings are aimed at researchers and teams building or fine-tuning reasoning models who have to choose between RL and SFT pipelines, and at anyone doing interpretability work on how reasoning models arrive at answers internally. It also speaks to teams evaluating how consistent or reliable a given reasoning model's compute usage is across repeated attempts at the same problem.

How to use it

The paper's methods, linear probing of hidden states and mean ablation across layers, offer a way to diagnose whether a given fine-tuning pipeline is producing well-structured, hierarchical representations versus flat, undifferentiated ones. The token-count variability analysis under repeated sampling can similarly be used as a diagnostic: high variability may flag a model whose reasoning policy is still under-determined rather than settled.

How solid is it

The claims rest on two converging methods, probe accuracy on hidden states and mean ablation importance by layer, plus a third independent analysis of token-count variability, which is a reasonable multi-angle approach. That said, the source text does not name specific models, model sizes, datasets, institutions, or authors, and does not give the underlying accuracy or variability numbers, so the strength of the effect cannot be quantified from what is available here; the paper also appears to be a preprint rather than confirmed as peer-reviewed.

Risks and caveats

The token-count variability finding is explicitly mixed, not uniform: some RL-tuned models showed high variability, others were just as consistent as SFT models, so the RL-versus-SFT distinction does not cleanly predict compute-allocation behavior on its own. Without model names, sample sizes, or numeric results in the available text, how far these representational findings generalize beyond the specific models studied is unclear.

“these findings demonstrate that RL training fundamentally restructures how models represent and process reasoning problems”

— the paper