PerceptionBench finds no MLLM tops 60% on atomic visual perception
Researchers have released PerceptionBench, a benchmark built to test the atomic visual perception of multimodal large language models (MLLMs) in isolation from reasoning or domain knowledge. The team argues that existing evaluations blur this distinction: holistic benchmarks mix up perceptual mistakes with reasoning failures, while narrower, application-specific benchmarks only cover fragmented domains shaped by ad hoc design choices, so neither approach can say cleanly whether a model's error came from seeing wrong or thinking wrong.
To fix this, the researchers took a bottom-up approach. They diagnosed the earliest point of failure in frontier MLLM responses across 42 existing benchmarks, then used those failure patterns to build an error taxonomy. The perception branch of that taxonomy defines ten atomic perceptual capabilities, and from it the team constructed 3,000 verified questions, each with a short, unambiguous answer and each isolating a single capability, so difficulty comes from perception itself rather than from reasoning or background knowledge.
Running the benchmark across sixteen frontier MLLMs showed that atomic perception is still largely unsolved: no model reached 60% accuracy. Perception-related hallucination came out as the weakest capability on average across the field. The results also showed that models with similar overall scores can have sharply different capability profiles underneath, meaning a single aggregate number can hide very different strengths and weaknesses in how a model actually sees. The authors present PerceptionBench as a capability-level standard for measuring and diagnosing where MLLMs' visual perception breaks down.
Key facts
- PerceptionBench isolates ten atomic visual perception capabilities, built from an error taxonomy derived by diagnosing failures across 42 existing benchmarks.
- The benchmark contains 3,000 verified, short-answer questions, each targeting a single perceptual capability rather than reasoning or knowledge.
- Across sixteen frontier MLLMs, no model reached 60% accuracy.
- Perception-related hallucination is the weakest capability on average among the tested models.
- Models with similar overall scores showed sharply divergent capability profiles once broken down by capability.
Why it matters
Most MLLM benchmarks report a single aggregate score, which conflates perception with reasoning and domain knowledge. PerceptionBench separates the two, and the result is that raw seeing, not higher-level reasoning, is exposed as a genuine and largely unsolved bottleneck: not one of sixteen frontier models cleared 60% accuracy on tasks built to be hard only because of perception.
Who it affects
MLLM developers and researchers who need to know whether a model's failures come from misperceiving an image or from downstream reasoning, and anyone building or selecting benchmarks for multimodal models, since the paper argues that existing holistic and application-specific benchmarks both fail to make this distinction cleanly.
How to use it
PerceptionBench is a 3,000-question evaluation set covering ten atomic perceptual capabilities. Model builders can run it to get a capability-level breakdown instead of one aggregate score, which the results show is necessary: two models can post similar overall numbers while differing sharply once perception is broken into its component capabilities.
How solid is it
The benchmark was built bottom-up from observed failure points across 42 existing benchmarks rather than from a hand-picked task list, and its 3,000 questions were verified and designed for short, unambiguous answers. It was evaluated against sixteen frontier MLLMs, giving the headline finding, that no model reaches 60% accuracy, a broad base of comparison across current models.
Risks and caveats
The source text does not name the sixteen models tested or give per-model scores, so it is not possible to say here which specific systems came closest to 60% or which lagged furthest behind. As with any benchmark, there is also the standard risk that scores on PerceptionBench specifically get optimized over time without perception genuinely improving elsewhere.