Glonce

SpatialScout benchmark finds VLM agents struggle to map 3D scenes

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact40Novelty55Relevance95Surprise35Players20Credibility70
SpatialScout benchmark finds VLM agents struggle to map 3D scenes

A new paper introduces SpatialScout, a benchmark for evaluating vision-language model (VLM) agents on active object-level 3D mapping. The authors start from the idea that spatial intelligence requires agents to actively gather visual evidence and merge their observations into a coherent representation of the three-dimensional world. Spatial question answering, they argue, offers limited evidence of whether models can build accurate metric maps through autonomous exploration.

The benchmark contains 80 diverse indoor scenes with 3,140 target objects in total. In each scene the agent receives an object inventory, RGB observations, camera calibration and camera poses. It then controls a camera to explore the scene and must estimate the center of each object in a shared coordinate frame. According to the authors, the task requires coordinating four things: viewpoint selection, cross-view association, geometric estimation and map revision, all under bounded interaction budgets.

Scoring combines localization accuracy and target coverage. It uses object-scale overlap scores that penalize missing predictions, so an agent cannot score well by locating only a few easy objects.

Experiments with frontier VLMs show substantial limitations. The strongest evaluated model, Claude Opus 5.5, achieves a mean overlap score of 41.71%. For that model, 47.23% of targets exceed an intersection-over-union (IoU) threshold of 0.5, but only 3.25% exceed 0.9. The authors read this as a substantial gap between recovering coarse object layouts and constructing complete, precise metric maps.

They present SpatialScout as a quantitative testbed for developing agents that build reliable 3D spatial representations through active perception. Code is available at https://github.com/JesseZZZZZ/SpatialScout.

Key facts

  • SpatialScout is a benchmark of 80 diverse indoor scenes containing 3,140 target objects, built to test VLM agents on active object-level 3D mapping.
  • Agents get an object inventory, RGB observations, camera calibration and camera poses, then control a camera and estimate object centers in a shared coordinate frame under bounded interaction budgets.
  • The strongest evaluated model, Claude Opus 5.5, reaches a mean overlap score of 41.71%.
  • For Claude Opus 5.5, 47.23% of targets exceed an IoU threshold of 0.5, but only 3.25% exceed 0.9.
  • The authors conclude there is a substantial gap between recovering coarse object layouts and building complete, precise metric maps; code is on GitHub.

Why it matters

Many tests of spatial ability for VLMs rely on question answering. The authors say that format gives limited evidence of whether a model can build an accurate metric map by exploring on its own. SpatialScout targets that gap by making the agent choose where to look, match objects across views, estimate geometry and revise its map. The headline result is the spread between coarse and precise localization: 47.23% of targets pass an IoU threshold of 0.5 for the best model, while 3.25% pass 0.9.

Who it affects

Researchers building agents that need to perceive and represent 3D space through active perception are the intended audience, since the authors offer the benchmark as a testbed for that work. The evaluated systems are frontier VLMs; the only one named is Claude Opus 5.5, as the strongest.

How to use it

The benchmark code is available at https://github.com/JesseZZZZZ/SpatialScout. In the task, an agent is given an object inventory, RGB observations, camera calibration and camera poses, controls a camera through each of the 80 scenes, and outputs estimated object centers in a shared coordinate frame. Results are scored with object-scale overlap scores that count missing predictions against the agent.

How solid is it

The account rests on the paper's abstract, which gives concrete figures: 80 scenes, 3,140 targets, a 41.71% mean overlap score, 47.23% of targets above IoU 0.5 and 3.25% above 0.9, all for Claude Opus 5.5. The abstract names no authors or institutions. Apart from Claude Opus 5.5, no other evaluated models are named and no scores for them are given. The 41.71% mean overlap score is not described as a comparison against any other model or method.

Risks and caveats

The mean overlap score and the IoU-threshold shares are different measures, so they should not be read as one number. The size of the interaction budget (number of steps or views) is not stated, and no human or non-VLM baseline is mentioned, so it is unclear how the 41.71% compares with anything beyond the paper's own framing of a gap. The results describe one benchmark of indoor scenes.

“These results reveal a substantial gap between recovering coarse object layouts and constructing complete, precise metric maps.”

— SpatialScout paper abstract