Glonce

AgentVQA benchmark: best of 22 vision-language models scores 58.5%

researchThe Glonce editorsHugging Face🔥56Glonce rating56Impact42Novelty55Relevance95Surprise28Players12Credibility75
AgentVQA benchmark: best of 22 vision-language models scores 58.5%

A paper on Hugging Face introduces AgentVQA, an offline visual question answering benchmark for testing vision-language models (VLMs) on realistic agentic scenarios. The authors start from a gap: VLMs handle a broad range of tasks, but their performance in agentic contexts is poorly understood. Online evaluations in simulators and real-world environments are essential for measuring how well interactive agents perform, but they are costly and hard to scale. Offline VQA benchmarks could allow faster and more reliable comparisons across agentic domains, the authors say, yet existing ones remain limited and fragmented.

AgentVQA covers five agent domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning. The authors built it by manually selecting 1000 challenging, real-world agent examples from 23 source datasets and adapting them so they can be scored reliably offline. They say the result preserves realistic scenarios from interactive environments across diverse domains, and that its model rankings track those from corresponding online evaluations.

The headline result is a low ceiling. Across the 22 evaluated models, the best achieves only 58.5% accuracy.

The authors also ran a cross-domain analysis of the errors. They conclude that mistakes are driven primarily by failures to ground and maintain accurate representations of the current and evolving visual state, rather than by task misinterpretation or downstream logical reasoning. The named failure types are localizing relevant UI elements, interpreting motion, tracking state changes over time, and inferring 3D spatial relationships.

Key facts

  • AgentVQA is an offline VQA benchmark for vision-language models in agentic settings, spanning five domains: Web Agents, Robotics, Egocentric Videos, Games, and Spatial Planning.
  • It contains 1000 manually selected, challenging, real-world agent examples drawn from 23 source datasets and adapted for reliable offline evaluation.
  • Across the 22 evaluated models, the best achieves only 58.5% accuracy.
  • The authors say AgentVQA model rankings track those from corresponding online evaluations.
  • Errors come mainly from failing to ground and maintain the current and evolving visual state (UI elements, motion, state changes over time, 3D spatial relationships), not from task misinterpretation or logical reasoning.

Why it matters

Testing agents in simulators and real environments is essential but costly and hard to scale. AgentVQA offers a cheaper offline alternative that the authors say ranks models the same way online evaluations do. The finding that even the best of 22 models reaches only 58.5% suggests current VLMs still struggle with agentic visual scenes. The error analysis points to a specific weak spot: tracking what is on screen and how it changes, not reasoning about it.

Who it affects

Researchers and developers building or evaluating vision-language models and agents in web navigation, robotics, egocentric video, games and spatial planning. Anyone choosing a VLM as the perception layer of an agent is the natural audience for these results.

How to use it

The benchmark is meant for fast offline comparison of VLMs across five agentic domains before or alongside costlier online tests. The abstract points to the weaknesses worth targeting: localizing UI elements, interpreting motion, tracking state changes over time and inferring 3D spatial relationships.

How solid is it

The setup is concrete: 1000 hand-selected examples, 23 source datasets, 22 models. The claim that rankings track online evaluations comes from the authors, and no correlation figure is given. The source does not say which models were tested, including the 58.5% leader. It also gives no per-domain accuracy or human baseline. This account rests on the paper's abstract alone.

Risks and caveats

A 1000-example set curated by hand reflects the authors' selection choices. Offline scoring is a proxy for interactive performance, and the authors themselves describe online evaluation as essential. The conclusion that errors are mostly perception failures rather than reasoning failures is the authors' own analysis.