Recursive language models beat chain-of-thought out of domain
The paper asks when restricting what a language model is allowed to see while it reasons actually helps it learn. It compares two setups. Standard chain-of-thought (CoT) lets the model read the entire reasoning trace built up so far before it takes the next step. Recursive language models (RLMs) are more restricted: each subtask is solved inside its own isolated context, cut off from the rest of the trace.
On data drawn from the same distribution as training, the restriction buys nothing. CoT can efficiently simulate the recursive rule from inside its own, more general hypothesis class, so its generalization guarantee only changes by a constant factor. Recursion is not needed there.
The difference shows up out of domain. Because CoT can see the full trace, it can fit the training data by leaning on context that sits outside the current subtask, a shortcut that works only as long as that surrounding context stays the same and breaks the moment those tokens change at test time. Recursive context isolation removes this option: with no outside context to lean on, the model has nothing left to fit but the actual subtask rule.
The authors stress that this is not a matter of CoT being unable to represent the correct rule. Its hypothesis class already covers the recursive rule as a special case. The failure is a simplicity bias: given a choice between the shortcut and the true rule, standard training picks the shortcut, because it is the simpler fit to the training distribution. The paper's conclusion is that covering the right rule inside a model's hypothesis class is not sufficient for it to actually learn that rule and reason correctly outside the training distribution, a result the authors present as standing in contrast to classical learning theory.
Key facts
- The paper compares standard chain-of-thought, which reads the full reasoning trace, against recursive language models, which solve each subtask in an isolated context.
- In-distribution, the two are roughly equivalent: CoT can simulate the recursive rule, so its generalization guarantee only changes by a constant factor.
- Out of domain, CoT can fit training data via a shortcut that relies on context outside the current subtask, and that shortcut breaks once the surrounding tokens change.
- Recursive context isolation rules out that shortcut by design, since there is no outside context left for the model to lean on.
- Even though CoT's hypothesis class already covers the correct recursive rule, simplicity bias makes standard training pick the shortcut over it, which the authors say contrasts with classical learning theory.
Why it matters
The paper gives a concrete theoretical reason why chain-of-thought reasoning can fail to generalize once inputs move away from the training distribution, even when the model is expressive enough to represent the correct rule. The problem is not capacity, it is which rule training ends up preferring. That reframes out-of-domain reasoning failures as a bias-in-training problem rather than a purely architectural one, and it gives isolating context between subtasks, as recursive language models do, a principled justification rather than treating it as just another prompting trick.
Who it affects
The result speaks to researchers designing multi-step or agentic reasoning systems that are expected to hold up on inputs unlike anything seen during training. The source is an arXiv abstract and does not name the authors, their institutions, or any specific model or system the argument was tested on.
How to use it
The paper is a theoretical argument, not a released tool or benchmark. The abstract does not describe an experimental setup, report accuracy numbers, or name specific model sizes, so there is nothing here to install or run; the practical takeaway is architectural: structuring a reasoning system so each subtask is solved in its own isolated context, rather than one long shared trace, is the mechanism the authors argue prevents the shortcut failure mode.
How solid is it
The source available here is the paper's abstract, which lays out the claim and its logical structure but does not itself contain experimental results, benchmark names, or numeric figures to back it. The argument is presented as a formal comparison of generalization guarantees between the two setups, but assessing how well it holds up would require reading the full paper.
Risks and caveats
No author names, institutional affiliations, publication venue, or timescale are given beyond the arXiv listing itself. No experiments, benchmarks, or numeric results appear in the abstract, so the claim should be read as a theoretical argument until the full paper's evidence can be checked.
“Even though CoT's class still covers the recursive rule, simplicity bias picks the shortcut over the truth.”
— the authors