Glonce

VA-Bench finds MLLMs struggle at embodied spatial reasoning

researchThe Glonce editorsHugging Face🔥68Glonce rating68Impact60Novelty65Relevance95Surprise50Players30Credibility85
VA-Bench finds MLLMs struggle at embodied spatial reasoning

Researchers introduce VA-Bench, a benchmark built to evaluate embodied spatial intelligence in general-purpose multimodal large language models (MLLMs) through a complete observe-reason-act-revise loop rather than simple object-location description. Under incomplete observation, a model must identify and acquire missing evidence, interpret it within a common spatial frame, and act on it. In VA-Bench, models learn procedural context from RGB-only demonstrations, actively select their own camera viewpoints, issue metric Cartesian commands, and revise those commands based on execution feedback, all without privileged object poses, oracle trajectories, or learned action heads; a fixed, model-agnostic controller executes only the targets the model itself specifies.

The benchmark contains 14 base task families (11 single-arm and three dual-arm), seven held-out geometry and layout variants, and a long-horizon track that composes five objects into one episode. The authors evaluate 12 primary model conditions, each run three independent times over the same 20 physically verified seeds per base task, and report terminal success alongside nine trajectory-level behavioral diagnostics and subtask progress.

The results show a sharp gap between component skills and end-to-end task success. In the annotated run, the best-performing model scores a perfect 100.0% on target localization and 78.9% on spatial relations, yet its three-run macro-average task success is only 53.93+/-3.17%. Active camera control matters: in one matched comparison, letting a model choose its own viewpoint rather than relying on passive multi-view observation raises task success from 27.86% to 57.50%. Generalization is fragile, too: transferring to the held-out geometry and layout variants can cut task success by over 30 percentage points. On the long-horizon, five-object composition track, no model completes a strict episode, despite showing substantial partial progress along the way.

Key facts

  • VA-Bench evaluates a full observe-reason-act-revise loop: models learn from RGB-only demonstrations, actively choose camera viewpoints, and issue metric Cartesian commands with no privileged object poses or oracle trajectories.
  • It covers 14 base task families (11 single-arm, three dual-arm), seven held-out geometry/layout variants, and a five-object long-horizon composition track, with 12 model conditions each run three times over 20 verified seeds per task.
  • The best model scores 100.0% on target localization and 78.9% on spatial relations, but its three-run macro-average task success is only 53.93+/-3.17%.
  • Active camera control raises task success from 27.86% to 57.50% over passive multi-view observation in a matched comparison; held-out geometric transfer can cut success by over 30 percentage points.
  • No model completes a strict long-horizon episode, even though models show substantial partial progress.

Why it matters

Most vision-language-action benchmarks measure single-step skills like grounding an object or predicting one action. VA-Bench instead scores the whole embodied loop: a model has to notice what it cannot see, move a camera to go find it, and then act on what it learns, revising as execution feedback comes in. The gap the paper documents, near-perfect perception scores next to barely-better-than-coin-flip task completion, is the kind of finding that reframes what "the model understands the scene" is worth in practice.

Who it affects

The benchmark targets researchers and engineers building or evaluating general-purpose MLLMs for robotic manipulation, especially teams working on active perception, viewpoint selection, and closed-loop control rather than open-loop, single-shot action prediction.

How to use it

VA-Bench is presented as an evaluation suite: 14 base task families plus seven held-out geometry/layout variants and a long-horizon composition track, run under a fixed, model-agnostic controller so that only model-specified targets are executed. The abstract does not state a release date, code availability, or dataset access.

How solid is it

The evaluation protocol is fairly rigorous for this kind of benchmark: 12 model conditions, each run three independent times over the same 20 physically verified seeds per base task, with results reported as macro-averages with variance (53.93+/-3.17%) rather than a single best-case number, plus nine separate trajectory-level behavioral diagnostics rather than terminal success alone.

Risks and caveats

The abstract does not name the best-performing model, so the headline numbers cannot be attributed to a specific system. Held-out geometric and layout transfer can reduce task success by over 30 percentage points, meaning strong scores on the base task set may not generalize to new scene geometry. No model completes a strict long-horizon, five-object episode, which limits how much the current results say about longer, more compositional real-world tasks.