Glonce

RoboSPA benchmark finds VLA models struggle with spatial reasoning, long-horizon planning

researchThe Glonce editorsHugging Face🔥71Glonce rating71Impact70Novelty75Relevance90Surprise45Players30Credibility85
RoboSPA benchmark finds VLA models struggle with spatial reasoning, long-horizon planning

Researchers have introduced RoboSPA (Robot Spatial-Procedural Assessment), a large-scale robotic manipulation dataset and benchmark built to diagnose how well Vision-Language-Action (VLA) models reason, not just whether they complete a task. The authors note that existing datasets and benchmarks mainly evaluate task completion under predefined settings, which gives limited insight into how a model's reasoning holds up as spatial and procedural complexity increases. RoboSPA targets that gap directly, organized around two dimensions: Fine-Grained Spatial Reasoning and Long-Horizon Procedural Planning. It spans 10 task categories and 56 base tasks, and each base task is instantiated across five difficulty levels, producing 280 task variants with increasing spatial ambiguity and procedural complexity. The underlying dataset comprises 527,000 collected trajectories, gathered across multiple robot embodiments and diverse scenes. Beyond a simple binary success rate, RoboSPA adds diagnostic metrics meant to give a more detailed read on model behavior than pass or fail alone.

Running experiments on representative VLA models against RoboSPA, the authors found that current systems still struggle with complex spatial relations, precise low-level execution, and memory-intensive planning. On that basis, they present RoboSPA as a challenging diagnostic benchmark for developing embodied agents that are more capable, reliable, and generalizable. The dataset and code are released publicly at https://github.com/fanzhenxuan/RoboSPA.

Key facts

  • RoboSPA covers 10 task categories and 56 base tasks, each instantiated across five difficulty levels for 280 total task variants.
  • The benchmark's dataset includes 527,000 collected trajectories across multiple robot embodiments and diverse scenes.
  • RoboSPA targets two dimensions specifically, fine-grained spatial reasoning and long-horizon procedural planning, rather than simple task completion.
  • Beyond a binary success rate, RoboSPA adds diagnostic metrics for a more detailed read on model behavior.
  • Experiments on representative VLA models found they still struggle with complex spatial relations, precise low-level execution, and memory-intensive planning.

Why it matters

Most VLA benchmarks measure whether a robot finishes a task under a fixed, predefined setup, which says little about how a model's reasoning holds up as a scene gets spatially ambiguous or a task gets procedurally longer. RoboSPA is built specifically to probe that reasoning rather than just count successes: it isolates two failure-prone dimensions, fine-grained spatial reasoning and long-horizon procedural planning, and steps up difficulty across five levels per task so a model's decline can be traced rather than just observed as a single pass or fail. That diagnostic structure, plus metrics beyond binary success rate, is what distinguishes it from a typical task-completion leaderboard.

Who it affects

Anyone building or evaluating Vision-Language-Action models for robotic manipulation gets a new diagnostic tool. Instead of a single success percentage, RoboSPA's structure, 10 task categories, 56 base tasks, five difficulty tiers and 280 variants total, lets a team see specifically whether their model's weak point is spatial precision, low-level execution, or long-horizon memory. Teams working toward real-world deployment, where scenes are cluttered and tasks span many steps, are the most direct audience, since those are exactly the conditions RoboSPA is designed to stress.

How to use it

RoboSPA's data and code are public at https://github.com/fanzhenxuan/RoboSPA; the source does not state a license, price, or access restriction. The benchmark is built from 527,000 collected trajectories across multiple robot embodiments and diverse scenes, organized into the 280 task variants, so a team can evaluate its own VLA model against the fixed task set and read out both the binary success rate and RoboSPA's added diagnostic metrics rather than building an evaluation protocol from scratch.

How solid is it

The source is a paper abstract page with no stated publication venue, peer-review status, or submission date, and it does not name individual authors or their institutional affiliations. The scale claims are specific and quantified, 56 base tasks, 280 variants, 527,000 trajectories, which is checkable in principle once the linked code and data are examined, but the abstract itself reports no numeric success-rate results for any named model, only the qualitative statement that current systems struggle. The finding rests on the authors' own experiments and framing; this text describes no independent replication.

Risks and caveats

The abstract does not identify which VLA models were tested, so it is unclear whether representative here means current state-of-the-art systems or a narrower selection the authors chose themselves. Without published success-rate numbers, the size of the gap RoboSPA exposes cannot be judged from this source alone, only that a gap exists. And because the benchmark's own authors are also the ones reporting that models struggle on it, independent evaluation would help confirm the difficulty holds up outside the paper's own experiments.

“Experiments on representative VLA models show that current systems still struggle with complex spatial relations, precise low-level execution, and memory-intensive planning.”

— the RoboSPA paper's authors