Qwen3-8B loses its step counter in 20 of 20 seeds when irrelevant text is added
Long-horizon failures of language models are usually reported as end-task success or as a horizon length. This arXiv paper takes a different route: it scores, step by step, three components of a 400-step ledger task in which every step is easy on its own and the model's own answers are fed back into its context. The three components are recognising a record already seen 5 to 300 steps earlier, processing the current record locally, and writing a counter that is incremented at every step.
The tests ran on Qwen3-8B in non-thinking mode, with 20 seeds, and were replicated on a second machine and backend. Every step was scored component by component. Two results hold.
First, duplicate-recognition errors rise late in the episode, from 0.0019 to 0.011 per step at risk across step 200. Over the same run the counter makes no error in 8,000 steps, and local processing is descriptively stable. The authors say the rise in duplicate errors combines long-distance recall errors and rising false alarms.
Second, a block of irrelevant content inserted before each step makes the counter fail in 20 of 20 seeds on each machine, against none without it. The effect of the same block on local processing is much smaller and not distinguishable from none.
Three pre-registered follow-ups then probe the counter. Its value depends on the value present in the response history: a corrupted value persists without correction in every seed. An exact external reminder, repeated before each message, re-aligns the counter within one to two steps. Under the irrelevant block, no counter error was observed when an exact compact reminder of the state, supplied externally by an oracle, preceded each message: 0 of 20 seeds had an error, against 15 without the reminder. This reminder also outperformed a length-matched neutral text. Whether a single reminder suffices was not conclusive, and the absence of damage to the other components was not established.
The authors close with a methodological point: an aggregate or exclusive error classification can hide the difference they report.
Key facts
- The 400-step ledger task is scored per step on three parts: recognising a record seen 5 to 300 steps earlier, local processing, and an incrementing counter. Tested on Qwen3-8B (non-thinking, 20 seeds), replicated on a second machine and backend.
- Duplicate-recognition errors rise from 0.0019 to 0.011 per step at risk across step 200, driven by long-distance recall errors and rising false alarms. The counter made no error in 8,000 steps.
- An irrelevant block inserted before each step breaks the counter in 20 of 20 seeds on each machine, against none without it; the effect on local processing is much smaller and not distinguishable from none.
- A corrupted counter value persists without correction in every seed, and an exact external reminder repeated before each message re-aligns it within one to two steps.
- With an oracle-supplied exact compact state reminder under the irrelevant block, 0 of 20 seeds had a counter error, against 15 without it.
Why it matters
Most long-horizon evaluations collapse behaviour into one number, such as end-task success or a horizon length. This paper argues that scoring components separately shows different failures with different causes. Here, recognising earlier records degrades slowly late in the episode, while the counter is either perfect or, with irrelevant content present, broken in every seed. The authors state that an aggregate or exclusive error classification can hide this difference.
Who it affects
The result is about one model, Qwen3-8B in non-thinking mode, on a synthetic ledger task. It is most relevant to researchers who measure long-horizon reliability and to anyone building loops in which a model's own earlier outputs are re-injected into its context, since the counter's value depended on what was in the response history.
How to use it
The source describes no tool, code or product. The practical finding it reports is that an exact external reminder of the state, repeated before each message, re-aligned a corrupted counter within one to two steps, and that under the irrelevant block an exact compact reminder supplied by an oracle outperformed a length-matched neutral text. The reminder came from an oracle, not from the model.
How solid is it
The setup is fairly careful: 20 seeds, per-step scoring of each component, replication on a second machine and backend, and three pre-registered follow-ups on the counter. The 20 of 20 failure result held on each machine, against none without the irrelevant block. This is a preprint abstract, and the claims are the authors' own.
Risks and caveats
Only Qwen3-8B (non-thinking) is tested, so the findings may not carry over to other models or sizes. The source does not state a mechanism for why irrelevant content breaks the counter. The reminder was supplied by an external oracle, and the source does not say it works without one. The authors report that whether a single reminder suffices was not conclusive, and that the absence of damage to the other components was not established. The zero-error result is an observation, not a general guarantee.
“An aggregate or exclusive error classification can hide the difference we report.”
— From the paper's abstract