Glonce

Grounded Action Models bring explicit 3D grounding to robot AI

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact70Novelty78Relevance92Surprise52Players30Credibility78
Grounded Action Models bring explicit 3D grounding to robot AI

Researchers propose Grounded Action Models (GAMs), a new paradigm for robot foundation models built around explicit 3D object grounding. The authors argue that manipulation policies need to know which objects matter and where they are in space, but the pretrained backbones behind current robot foundation models, whether language in vision-language-action models (VLAs) or video generation in world-action models (WAMs), do not directly require this kind of metric grounding. Instead, it is left to be learned implicitly from robot demonstrations. GAM addresses this by accepting language, point, or box prompts, converting them into a shared object-centric representation of the selected objects. That representation captures both target-focused visual features and metric object geometry, and is combined with a history of robot states through a multi-stream transformer to predict chunks of actions. GAM can run on its own, or act as a low-level controller directed by a higher-level planner that issues commands through various input modes, which the authors say enables long-horizon, memory-dependent manipulation tasks. On the RoboTwin 2.0 simulation benchmark, GAM reaches an average success rate of 55.3% across 50 tasks, ahead of Spatial Forcing's 52.0%, despite its action policy being trained only on clean-scene demonstrations. Under scene randomization on the same benchmark, GAM scores 47.6% versus 30.4% for Abot-M0. On LIBERO-PRO, GAM posts a state-of-the-art average success rate of 61% across 16 perturbation settings, compared with 53% for π_{0.5}, with its biggest gains coming when target objects are relocated or newly designated. On real hardware, GAM keeps 17 of 20 successes under visual shift on a bimanual YAM robot, against 4 of 20 for π_{0.5}. Paired with a Molmo2 high-level planner on a Franka arm, the system completes 64.7% of steps in-distribution and 49.8% out-of-distribution on long-horizon, memory-dependent tasks.

Key facts

  • GAM converts language, point, or box prompts into a shared object-centric 3D representation, mixed with robot state history via a multi-stream transformer to predict action chunks.
  • On RoboTwin 2.0 (50 tasks), GAM scores 55.3% average success versus 52.0% for Spatial Forcing, and 47.6% versus 30.4% for Abot-M0 under scene randomization.
  • On LIBERO-PRO's 16 perturbation settings, GAM reaches a state-of-the-art 61% average success rate versus 53% for π_{0.5}.
  • On a real bimanual YAM robot under visual shift, GAM succeeds 17 out of 20 times versus 4 out of 20 for π_{0.5}.
  • Combined with a Molmo2 planner on a Franka arm, GAM completes 64.7% of steps in-distribution and 49.8% out-of-distribution on long-horizon, memory-dependent tasks.

Why it matters

Most robot foundation models inherit their perception from backbones built for other jobs: language understanding in vision-language-action models, or video generation in world-action models. Neither of those backbones is required to represent where objects actually are in 3D space; that grounding is left to emerge indirectly from watching robot demonstrations. GAM's proposal is to make that grounding explicit and shared across every prompt type, language, points, or boxes, so the policy always has a metric, object-centric representation to act on rather than an implicit one it had to infer.

Who it affects

The work targets researchers building robot foundation models and manipulation policies, particularly teams working on generalization across cluttered or randomized scenes and on long-horizon tasks that require memory. It is a research architecture rather than a deployed product, and the source gives no timeline for real-world use beyond the experiments described.

How to use it

GAM accepts three prompt types, language, points, or bounding boxes, all mapped into the same object-centric representation before being fused with robot state history. It can act as a complete, autonomous policy, or as a low-level controller that a separate high-level planner drives through whichever input mode fits the situation; the authors demonstrate this pairing with a Molmo2 planner controlling a Franka arm on long-horizon, memory-dependent tasks.

How solid is it

The claims rest on comparisons across two simulation benchmarks and two real robots. On RoboTwin 2.0's 50 tasks, GAM's 55.3% average success rate beats Spatial Forcing's 52.0%, and its edge widens under scene randomization, 47.6% versus 30.4% for Abot-M0, even though GAM's action policy was trained only on clean scenes. On LIBERO-PRO's 16 perturbation settings it posts a state-of-the-art 61% against 53% for π_{0.5}, with the largest gains when target objects are moved or reassigned. The real-robot results point the same way: 17 of 20 successes under visual shift on a bimanual YAM versus 4 of 20 for π_{0.5}, and 64.7% in-distribution versus 49.8% out-of-distribution step completion when paired with a Molmo2 planner on a Franka arm.

Risks and caveats

The source text gives no institutional affiliation for the authors, no publication venue or submission date, and no statement on whether code, models, or data will be released, so none of that can be confirmed here. The in-distribution/out-of-distribution split (ID/OOD) reported for the Franka results is not itself spelled out in the text. All of the reported gains are against a specific set of baselines and benchmarks named in the paper; how GAM performs outside those settings is not addressed.