Paper finds most wrong actions by fast agent models follow a correct reading of the state
The paper, titled "Reading Right, Acting Wrong", looks at "System One" models: models that choose an agent's next action by scoring a supplied set of options in a single pass. That makes them fast and cheap. The problem is that when they choose wrongly, there is nothing to inspect. Did the model misread what the agent has seen, or did it read the state correctly and still act wrongly?
The authors answer this behaviourally. They use a frozen benchmark of agent decision states drawn from simulated environments, customer-service policies, tool use, web navigation and coding. Every next-action question is paired with factual probes about the same state, and each probe is answered independently. That lets them separate errors of reading from errors of acting.
Across seventeen fast systems and one reasoning model, the answer is consistent: most wrong actions follow correct reading, and the more accurate a system is, the larger the share of its errors that take this form. These errors are not noise. They gather on the same states across systems of every kind.
The paper also tests a case where the probed facts provably determine the right action. Models that read every fact still fail to combine them into the correct choice, but when their own answers to the probes are composed by code, the failure is repaired. On real states, the paper adds, what is missing is often not a fact but the next sub-goal.
Key facts
- System One models pick an agent's next action by scoring a supplied set of options in a single pass, which makes them fast and cheap but leaves nothing to inspect when they err.
- The benchmark is frozen and covers decision states from simulated environments, customer-service policies, tool use, web navigation and coding; each next-action question is paired with independently answered factual probes.
- Across seventeen fast systems and a reasoning model, most wrong actions follow correct reading, and the more accurate a system, the more of its errors take this form.
- The errors cluster on the same states across systems of every kind, so they are not noise.
- Where the probed facts provably determine the action, composing the models' own probe answers by code repairs the failure; on real states, the missing piece is often the next sub-goal.
Why it matters
Fast single-pass action selection is attractive because it is cheap, but a wrong choice gives no trace to debug. This paper separates two very different failure modes, misreading the state and misusing a correct reading, and reports that the second dominates. If that holds, improving how a model perceives the state will not fix most of its mistakes.
Who it affects
Anyone building or evaluating agents that choose actions from a supplied list of options in one pass. The benchmark spans customer-service policies, tool use, web navigation and coding, so the finding is framed across several kinds of agent work rather than one domain.
How to use it
The practical idea is the method: ask factual questions about the same state alongside the next-action question, answer each independently, and compare. The paper also reports that, where the facts provably determine the action, composing the models' own answers by code repairs the failure. No code or dataset availability is mentioned in the abstract.
How solid is it
This is a preprint listed on Hugging Face Papers, and the evidence rests on a frozen benchmark tested across seventeen fast systems and a reasoning model. The finding is described as consistent across them. The abstract names no authors or institutions, and no numeric rates of errors that follow correct reading are given; only "most".
Risks and caveats
The abstract does not name the systems tested, the benchmark, the number of decision states or probes, or any accuracy figures, so the size of the effect cannot be judged from it. Only one reasoning model is included, which limits what can be said about reasoning systems. The code-composition repair is shown where the facts provably determine the action; on real states the paper notes that what is missing is often the next sub-goal rather than a fact.
“These errors are not noise; they gather on the same states across systems of every kind.”
— Abstract of "Reading Right, Acting Wrong"