SO-Arena benchmark finds best System One model ranks only seventh in reliability

A new paper introduces SO-Arena, a benchmark for System One models. These are models that promise fast, general-purpose decision-making without lengthy token generation. The authors argue that understanding what such models can do, and how far their decisions can be trusted, takes more than aggregate scores on current benchmarks, which often give limited insight into the breadth of capabilities and the reliability of decisions.
SO-Arena covers 11 domains and five capability families. For the capability spectrum, the authors describe three features. First, evidence-grounded evaluation: evidence comes from reliable sources, and reference labels are manually verified against the inputs the model actually sees. Second, fine-grained capability assessment through domain-specific difficulty criteria. Third, realistic interactive evaluation, where models iteratively execute actions in sandboxes. Reasoning-intensive tasks also allow a direct system one-two comparison.
Beyond accuracy, SO-Arena measures decision reliability in two complementary ways. One is decision stability under controlled changes to the set of options. The other is missing-answer recognition: whether a model notices when no valid option is available.
The authors evaluated 21 models. Two findings stand out. The benefits of scaling and of additional reasoning vary substantially across domains. And strong overall performance does not necessarily imply reliable decision-making: the best-performing System One model ranks only seventh in reliability. The authors conclude that capability spectrum and decision reliability should be characterised as complementary dimensions, giving a systematic view of the strengths and limitations of System One models.
Key facts
- SO-Arena is a benchmark for System One models, which aim for fast decisions without lengthy token generation, covering 11 domains and five capability families.
- It measures decision reliability beyond accuracy: stability when the set of options is changed in a controlled way, and recognition of cases where no valid option exists.
- Capability tests include interactive tasks in which models iteratively execute actions in sandboxes, and reference labels are manually verified against model-visible inputs.
- Across 21 evaluated models, the best-performing System One model ranks only seventh in reliability.
- The benefits of scaling and additional reasoning vary substantially across domains.
Why it matters
Fast decision-making models are pitched as general-purpose, yet the authors say aggregate scores on current benchmarks reveal little about the breadth of what they can do or how dependable their decisions are. The headline result makes the point concretely: the model that performs best overall ranks only seventh when reliability is scored. A single leaderboard number can hide that gap.
Who it affects
Researchers building or comparing System One models are the direct audience, since SO-Arena offers a way to profile capability by domain and to test reliability separately from accuracy. Anyone choosing a fast-decision model for a task would also be affected, because the findings suggest the top performer is not automatically the most reliable one.
How to use it
The abstract describes the benchmark's design: 11 domains, five capability families, sandboxed interactive tasks, and two reliability measures (option-set stability and missing-answer recognition). Its reasoning-intensive tasks allow a system one-two comparison. No release date, code or dataset availability is stated, so there is nothing yet to say about running it.
How solid is it
The material is a paper abstract, so the claims are the authors' own and the underlying results are not visible here. The abstract reports 21 models evaluated and a single concrete result, the seventh-place reliability rank of the best-performing System One model. No accuracy scores, reliability scores or other numeric results are given beyond that rank. The authors say reference labels were manually verified and evidence drawn from reliable sources.
Risks and caveats
No model names are given, neither the best-performing System One model nor any of the other 20 evaluated. The five capability families and the 11 domains are not listed. It is not stated whether the best model is best in accuracy or in an aggregate score; the text says only best-performing and strong overall performance. The abstract also does not define System One beyond fast decision-making without lengthy token generation, and it does not say which models count as System One versus System Two.
“strong overall performance does not necessarily imply reliable decision-making: the best-performing System One model ranks only seventh in reliability”
— SO-Arena paper abstract