Glonce

Paper finds RL rarely invents 'wait' in reasoning models

researchThe Glonce editorsHugging Face🔥60Glonce rating60Impact45Novelty60Relevance100Surprise35Players10Credibility75
Paper finds RL rarely invents 'wait' in reasoning models

Reasoning models often write the word 'wait' before they reconsider a step. Reinforcement learning (RL) is commonly credited with teaching them to do this. A paper listed on Hugging Face argues that this view can confuse what RL discovers with what a model inherits from its training data, and notes that large RL accuracy gains do not require the word.

To test it, the authors trace the word through the training histories of open models. They read it with label-free indices of frequency, position, form and usage, plus an injected-error test, all computed from sampled outputs.

Their main finding is that zero-RL (RL applied directly to a base model) almost never coins the marker. On base models that lack the word, RL raises MATH-500 accuracy substantially while 'wait' stays near absent. Instead, the word arrives with training data: through mid-training in OLMo-3, and through fine-tuning on reasoning traces, which installs it in most models whose base lacks it.

When the base model already uses the word, the picture depends on the recipe. Both released RL-Zero recipes, from OLMo-3 and MiMo, multiply it. KL-free GRPO from the same base does not. The OLMo-3 recipe also wears down the word's form and makes it a more sensitive but more liberal alarm. RL applied after supervised fine-tuning barely changes the marker, and the authors' own GRPO recipe erases a marker that fine-tuning had seeded.

The paper ends with two cautions. Marker frequency alone does not indicate RL-induced reasoning. And the authors warn against LLM-as-a-judge revision metrics, because automated judges can reverse human-labelled trends.

Key facts

  • On base models that lack the word, RL raises MATH-500 accuracy substantially while 'wait' stays near absent, so zero-RL almost never coins the marker.
  • The word arrives with training data: mid-training in OLMo-3, and fine-tuning on reasoning traces installs it in most models whose base lacks it.
  • Where the base already uses 'wait', both released RL-Zero recipes (OLMo-3, MiMo) multiply it; KL-free GRPO from the same base does not.
  • RL after supervised fine-tuning barely changes the marker, and the authors' GRPO recipe erases a marker seeded by fine-tuning.
  • Marker frequency alone does not indicate RL-induced reasoning, and the authors caution against LLM-as-a-judge revision metrics.

Why it matters

'Wait' and similar self-correction words are often read as a sign that RL taught a model to reconsider its work. The paper says that reading can mix up what RL discovers with what a model inherits from its training data. If the word mostly comes from mid-training or fine-tuning on reasoning traces, then seeing it in a model's output says little about what RL contributed. The paper also notes that large RL accuracy gains do not require the word.

Who it affects

Mainly people who train or evaluate reasoning models, including those working with open recipes such as OLMo-3 and MiMo, and anyone who uses the frequency of reflection words as evidence that a model has learned to revise itself. Teams that fine-tune on reasoning traces are also relevant, since the paper says that step installs the word in most models whose base lacks it.

How to use it

The practical takeaways come straight from the authors' cautions. Do not treat how often 'wait' appears as proof of RL-induced reasoning. Be wary of LLM-as-a-judge revision metrics, since automated judges can reverse human-labelled trends. The authors' own tools are label-free indices of frequency, position, form and usage, plus an injected-error test, all computed from sampled outputs. No code, dataset or model release is mentioned.

How solid is it

The source is the paper's abstract as listed on Hugging Face, so the findings are the authors' own claims. They are stated qualitatively, with words like 'substantially' and 'near absent'. No numeric accuracy values, frequencies or effect sizes are given. The abstract does not say which specific models besides OLMo-3 and MiMo were examined, nor how many.

Risks and caveats

The paper does not say RL never matters for reasoning. It says the word is not required for large RL accuracy gains and that frequency alone does not indicate RL-induced reasoning. Results differ by recipe: the released OLMo-3 and MiMo RL-Zero recipes multiply the word when the base already has it, while KL-free GRPO does not, so findings about one recipe should not be assumed to carry over to another. The detail behind each claim sits in the full paper.

“This view can confuse what RL discovers with what a model inherits from its training data.”

— Paper abstract, Hugging Face papers listing