Glonce

HumanCLAW benchmark finds vision-language models fail at embodied tasks

researchThe Glonce editorsHugging Face🔥57Glonce rating57Impact65Novelty78Relevance92Surprise60Players40Credibility85
HumanCLAW benchmark finds vision-language models fail at embodied tasks

Researchers led by Siyao Li and co-authors built HumanCLAW, an evaluation framework meant to answer a specific problem: when a robot or embodied agent fails a task, it is hard to tell whether the vision-language model (VLM) made a bad decision or whether the body's motor controller simply failed to execute it, for example by losing balance and falling. HumanCLAW separates these two causes. At every step, an off-the-shelf VLM is harnessed to issue a single atomic skill command (such as a short instruction about what to do next), and that command is translated into a sub-second chunk of continuous full-body motion that plays out with real physical consequences, including gravity and collisions. Because the body can act freely in a physically simulated world while balance and motor-control errors are factored out of the scoring, what remains measurable is the VLM's own action intelligence: its moment-to-moment choice of what the body should do next, independent of whether the body's own execution succeeds.

On top of this framework the authors built HumanCLAW-Bench, a benchmark of 1,218 long-horizon, egocentric episodes that require finding a target, navigating to it and interacting with it, spread across 41 indoor scenes. They tested nine state-of-the-art VLMs against it. None solved the benchmark; the best-performing model reached only a 16.8% success rate. The authors report that recognizing the target object or location is not what holds these models back. Instead, the failure mode is a lack of embodied self-awareness: the models lose track of their own body's state, unable to reliably tell where the body currently is, whether it has already reached the goal, or whether it has collided with an obstacle.

Key facts

  • HumanCLAW decouples a VLM's action decisions from motor execution by translating each atomic skill command into a sub-second chunk of physically simulated full-body motion.
  • HumanCLAW-Bench comprises 1,218 long-horizon, egocentric find-navigate-interact episodes across 41 indoor scenes.
  • Nine state-of-the-art VLMs were tested; none solved the benchmark, and the best model achieved only a 16.8% success rate.
  • The bottleneck is not target recognition but embodied self-awareness: models lose track of their own body's position and state.
  • Failures include not knowing whether the goal has been reached or whether the body has hit an obstacle.

Why it matters

Most embodied-AI benchmarks conflate two separate failures: a model choosing the wrong action, and a body failing to physically carry it out. HumanCLAW isolates the first cause by handing execution to a physics-based body simulation with real gravity and collisions, so a low score can be attributed to the VLM's decision-making rather than clumsy motor control. That framing turns a robotics engineering problem into a clean test of whether current VLMs can reason about acting in a physical world at all, and the answer, a 16.8% best-case success rate across nine leading models, suggests they largely cannot yet.

Who it affects

The result concerns anyone building or evaluating embodied agents and household or navigation robots that plan to use general-purpose VLMs as their decision-making core, as well as researchers designing future embodied benchmarks, since HumanCLAW's decoupling approach offers a template for isolating decision quality from execution quality.

How to use it

The paper does not describe pricing, licensing or a product release; it presents an evaluation framework and benchmark (HumanCLAW-Bench) intended for researchers to test VLMs against, not a tool for end users.

How solid is it

The claims rest on a fairly large, purpose-built benchmark, 1,218 episodes across 41 scenes, tested against nine current state-of-the-art VLMs, with a physics-based execution layer meant to control for motor-side confounds. The text does not name which nine models were tested or give per-model scores beyond the best result of 16.8%, so the breadth of failure across specific model families cannot be assessed from this summary alone.

Risks and caveats

The 16.8% figure describes the single best-performing model; the text does not state how the other eight models compared, so the spread of performance is unclear. The benchmark's indoor, egocentric find-navigate-interact setup may not generalize to other embodied task types, and the diagnosis of 'embodied self-awareness' as the core deficit is the authors' own interpretation of the failure pattern rather than an independently verified mechanism.