Glonce

Moonshot AI's PerceptionBench shows no frontier model tops 60% on pure vision

researchThe Glonce editorsThe Decoder🔥77Glonce rating77Impact70Novelty70Relevance100Surprise65Players70Credibility80
Moonshot AI's PerceptionBench shows no frontier model tops 60% on pure vision

Moonshot AI, the team behind the Kimi AI assistant, has released PerceptionBench, a benchmark built to test how well multimodal AI models can actually see, isolated from logical reasoning and outside knowledge. Every question in it can be answered by looking at the image alone. The authors built the benchmark's taxonomy from real model errors rather than from theory: after analyzing 42 open-source vision benchmarks and finding little overlap in what errors each one caught, they traced failures back to their earliest failed step and grouped them into ten skill domains: Visual Relation, Counting, Attributes, Depth & 3D, Localization, Comparison, Fine-grained Recognition, Context Integration, OCR, and Hallucination. From an internal pool of over 17,000 verified questions, Moonshot AI is publishing 3,000 tasks, 60 percent derived from attributed model errors and 40 percent reformulated using augmented images. The tasks look trivial on the surface, such as locating a symbol on a clock face or counting flowers inside a red box, but among the 16 frontier models tested, none broke 60 percent overall accuracy. GPT-5.6 Sol led at 59.7 percent, followed by Kimi K3 at 58.5 percent, Claude Fable 5 at 57.2 percent, Gemini 3.1 Pro at 56.2 percent, and GPT-5.5 at 55.8 percent. Open-source models trailed further behind: Qwen3.5-397B-A17B scored 47.5 percent and GLM-4.6V scored 32.5 percent. The category-level breakdown proved more revealing than the overall ranking, since models with near-identical aggregate scores diverged sharply on individual skills. Hallucination, the sub-test that checks whether a model invents objects that are not there when the correct answer is simply zero, came out as the weakest skill on average. GPT-5.6 Sol, despite topping the overall leaderboard, scored just 26.9 percent on hallucination, while the otherwise weaker Gemini 3.5 Flash scored 50.6 percent there. The authors argue that many failures usually blamed on reasoning actually originate at the perception stage: when a model botches a multi-step task, the first step of correctly reading the image has often already gone wrong. PerceptionBench splits those multi-step questions into perception-only sub-questions to pinpoint exactly which visual ability failed, and the dataset and evaluation code are published on GitHub under MoonshotAI/PerceptionBench. The article situates the result alongside two earlier studies. Moonshot AI's own WorldVQA benchmark, which also separates object recognition from reasoning, found the best model, Gemini 3 Pro, fell short of 50 percent at 47.4 percent, with all models systematically overestimating their own confidence. A separate study by Chinese institutions involving Moonshot AI, using a benchmark called BabyVision built around tasks tied to early childhood development such as tracing lines or counting hidden blocks, found Gemini 3 Pro scored 49.7 percent against a human score of 94.1 percent; the researchers attribute the gap to a verbalization bottleneck, where visual information gets translated into language and loses fidelity along the way.

Key facts

  • Moonshot AI released PerceptionBench, a benchmark that isolates visual perception from reasoning and knowledge, built from ten atomic skill domains derived from real model errors across 42 existing benchmarks
  • Among 16 frontier models tested, none reached 60 percent accuracy: GPT-5.6 Sol led at 59.7 percent, ahead of Kimi K3 (58.5 percent), Claude Fable 5 (57.2 percent), and Gemini 3.1 Pro (56.2 percent)
  • Hallucination, detecting when the correct answer is that an object simply is not in the image, is the weakest skill on average; GPT-5.6 Sol scored only 26.9 percent there despite ranking first overall
  • The authors conclude many errors attributed to flawed reasoning actually originate at the image-reading stage, before any reasoning happens
  • Related benchmarks WorldVQA and BabyVision show the same pattern: Gemini 3 Pro scored 47.4 percent on WorldVQA and 49.7 percent on BabyVision, where humans hit 94.1 percent

Why it matters

Multimodal AI systems are routinely deployed on the assumption that if they answer a question wrongly, the fault lies in reasoning, the part vendors keep tuning and boasting about. PerceptionBench challenges that assumption directly by proving that a large share of errors happen earlier, at the point where the model reads the image, before any reasoning is applied. That reframes where the industry's real bottleneck sits: closing the reasoning gap between models does little good if the underlying visual read is already wrong.

Who it affects

Anyone building or evaluating multimodal AI products, including the labs behind GPT-5.6 Sol, Kimi K3, Claude Fable 5, and Gemini 3.1 Pro, all of which were benchmarked and all of which underperformed. It also matters to teams deploying these models for tasks that depend on accurate visual reading, such as counting, spatial layout, or fine-grained recognition, since the benchmark's category breakdown shows aggregate scores can mask severe, model-specific weak points.

How to use it

Moonshot AI has published the PerceptionBench dataset (3,000 of the more than 17,000 verified questions in its internal pool) and its evaluation code on GitHub under MoonshotAI/PerceptionBench, so teams can run their own models against the same ten skill domains and compare category-level results rather than relying on a single aggregate score.

How solid is it

The benchmark's taxonomy was built empirically: the authors analyzed 42 existing open-source vision benchmarks, found little overlap in the errors each one caught, and traced failures back to their earliest failed step to define the ten skill domains, rather than defining categories from theory first. The 16-model comparison and the category-level breakdown come directly from Moonshot AI's own published results. The finding is also corroborated by two related studies cited in the piece, WorldVQA and BabyVision, both of which independently found frontier models falling well short of full accuracy on tasks that isolate perception from reasoning.

Risks and caveats

PerceptionBench and the two corroborating studies (WorldVQA, BabyVision) all involve Moonshot AI, which also builds the Kimi K3 model that performs competitively in the results, so the benchmark's framing and category weighting were not set by an independent third party. The article gives no individual named authors, institutional affiliations beyond "Moonshot AI," or a publication date for PerceptionBench, and no sample size or model count is stated for the BabyVision study.