TuringSpace benchmark tests multimodal LLMs on 125 interactive CAPTCHAs

A paper on Hugging Face introduces TuringSpace, a synthetic web environment built to test whether multimodal large language models (MLLMs) can act on what they see. The authors start from how humans work in the visual world: we infer the 3D layout behind a 2D view, predict how an action will change it, and correct ourselves after seeing the result. CAPTCHAs, they note, are a natural test because people solve them through exactly this loop, clicking and dragging on the screen and adjusting after each result.
The authors argue that existing CAPTCHA benchmarks are largely solved and rarely spatial, while spatial benchmarks rarely require action. TuringSpace is their answer: 125 interactive CAPTCHAs spanning five spatial abilities, which must be solved through clicks and drags on the screen. Every task is a program that generates new instances, with a checker and an oracle, so the tasks also work as a scalable training gym.
The results show a wide gap between humans and models. Humans solve 94.1% of the tasks. The best model, GPT-6 Astra, still fails 25.9% of them. The best open-weight model, Kimi K3, fails 68.0%.
The authors' analysis of the failures points to two patterns. Most failures come from misreading the screenshot. And model reasoning rarely corrects a wrong belief but instead hallucinates evidence to defend it.
To test whether such spatial abilities can be learned, the authors fine-tuned Qwen3.5-9B on the gym. Its success rate rose from 1.6% to 33.1%, above Kimi K3. It also improved on all seven out-of-distribution benchmarks, including a 9.9-point gain on VSI-Bench, which uses real-world indoor videos. The authors conclude that grounding spatial intelligence in screen actions exposes what MLLMs lack and offers a scalable way to teach it, with gains that transfer to real-world scenes. They say they will release TuringSpace to support further study.
Key facts
- TuringSpace is a synthetic web environment of 125 interactive CAPTCHAs spanning five spatial abilities, solved through clicks and drags.
- Humans solve 94.1% of the tasks; the best model, GPT-6 Astra, fails 25.9%, and the best open-weight model, Kimi K3, fails 68.0%.
- The authors' analysis finds most failures come from misreading the screenshot, and that model reasoning rarely corrects a wrong belief but hallucinates evidence to defend it.
- Fine-tuning Qwen3.5-9B on the gym lifts its success rate from 1.6% to 33.1%, above Kimi K3, with gains on all seven out-of-distribution benchmarks, including 9.9 points on VSI-Bench.
- Every task is a program that generates new instances with a checker and an oracle, so it doubles as a scalable training gym; the authors say they will release TuringSpace.
Why it matters
Many MLLM tests ask a model to describe or answer questions about an image. TuringSpace asks it to act: infer layout from a 2D view, predict what an action changes, and correct itself after the result. The authors say existing CAPTCHA benchmarks are largely solved and rarely spatial, while spatial benchmarks rarely require action, so this fills a gap between the two. The headline gap is stark: humans succeed on 94.1% of tasks, while even the best model, GPT-6 Astra, fails 25.9%. The fine-tuning result adds a second point: the weakness can be trained on, and the gains carry over to real-world indoor videos.
Who it affects
Researchers building and evaluating multimodal models, especially those working on spatial reasoning and agents that operate screens through clicks and drags. Teams working with open-weight models are also in scope: the best one tested, Kimi K3, fails 68.0% of tasks, while a 9B-parameter model fine-tuned on the gym, Qwen3.5-9B, reaches a 33.1% success rate, above Kimi K3.
How to use it
For now this is a paper, not a product. The authors say they will release TuringSpace to support further study. Because every task is a program that generates fresh instances with a checker and an oracle, it is designed to serve both as an evaluation set and as a training gym once it is available. No release date, URL or licence for TuringSpace is given.
How solid is it
The numbers come from the paper's own abstract and are self-reported by the authors; no independent replication is described. The comparison is clean on the headline figures: 125 tasks, 94.1% human success, 25.9% and 68.0% failure for GPT-6 Astra and Kimi K3. Several details a reader would want to check are not given: no details of the fine-tuning method (training data size, RL vs SFT, compute) are given, and the names or scores of the other six out-of-distribution benchmarks are not given; only the 9.9-point gain on VSI-Bench is.
Risks and caveats
The success rates of GPT-6 Astra and Kimi K3 are not stated directly; only their failure rates are, so avoid converting them without care. The five spatial abilities are not listed. The claim that failures mostly come from misreading the screenshot, and that reasoning hallucinates evidence to defend a wrong belief, is the authors' own analysis. The 33.1% result for the fine-tuned model is still far below the 94.1% human rate. The release of TuringSpace is a stated intention, not yet a fact.
“model reasoning rarely corrects a wrong belief but hallucinates evidence to defend it”
— TuringSpace authors, on their failure analysis