Glonce

World Action Agent lets VLMs pilot robots, hits 75.6% on LIBERO-Pro

researchThe Glonce editorsHugging Face🔥75Glonce rating75Impact65Novelty80Relevance98Surprise60Players30Credibility85
World Action Agent lets VLMs pilot robots, hits 75.6% on LIBERO-Pro

Researchers describe World Action Agent (WAA), a multi-agent harness through which general-purpose vision-language models (VLMs) pilot robots with basic tools, making every decision inside what the authors call a visual action workspace. They argue that existing systems either use VLMs indirectly, to predict constraints or write programs, or give them only a view of the scene rather than a world they can actually act in. WAA's workspace has three properties: contact views, chosen automatically from the scene geometry, show the area around the current interaction; action rehearsal turns each action into an editable proposal that the agent, alone or with help from a separate Imagination Agent, previews and revises against planning feedback before it is executed; and in-view correction closes the loop between observation, rehearsal, and low-level execution, letting the agent remove residual offsets in the same view where it spots them. Through this workspace, WAA builds embodied procedural knowledge two ways: it evolves multimodal skills from expert videos and human demonstrations under evidence-based review, calling on them through a dedicated Skill Agent, and its own interaction traces are used to train smaller VLMs to operate the same harness. On the LIBERO-Pro benchmark, WAA using skills evolved only from the LIBERO-90 dataset reaches a state-of-the-art 75.6% average success rate, beating end-to-end vision-language-action (VLA) models, code-as-policy agents, and a visual-harness baseline built on the same backbone model. The same skills, evolved only on LIBERO-90, remain effective on the separate robosuite benchmark without any further learning. Separately, fine-tuning a smaller model, Qwen3.5-9B, on traces of the harness's own interactions raises its out-of-domain success rate from 1.7% before fine-tuning to 43.3% after.

Key facts

  • World Action Agent (WAA) is a multi-agent harness that has VLMs act inside a visual action workspace instead of only observing a scene.
  • The workspace combines automatically selected contact views, editable action rehearsal previewed by the agent or an Imagination Agent, and in-view correction that removes residual offsets.
  • WAA reaches a state-of-the-art 75.6% average success on LIBERO-Pro using skills evolved only from LIBERO-90, beating end-to-end VLAs, code-as-policy agents, and a matched visual-harness baseline.
  • Skills evolved on LIBERO-90 transfer to the robosuite benchmark without further learning.
  • Fine-tuning the smaller Qwen3.5-9B model on WAA's own interaction traces lifts its out-of-domain success rate from 1.7% to 43.3%.

Why it matters

Most VLM-based robot systems either keep the model at arm's length, using it to predict constraints or write control programs, or let it merely look at a scene. WAA instead has the VLM make every manipulation decision inside an editable, previewable action workspace, which the authors present as a more direct way to put a general-purpose model's spatial reasoning to work in physical manipulation.

Who it affects

The work targets robotics and embodied-AI researchers building manipulation systems on top of general-purpose vision-language models, as well as teams evaluating VLA models and code-as-policy agents on standard benchmarks like LIBERO-Pro and robosuite.

How to use it

The source text describes a research harness and benchmark results rather than a released product; it does not mention any code or dataset release, licence, or pricing, so there is nothing yet to deploy directly.

How solid is it

The claims come from the paper's own benchmark comparisons against end-to-end VLAs, code-as-policy agents, and a matched visual-harness baseline on LIBERO-Pro, plus a transfer test on robosuite and a fine-tuning experiment with Qwen3.5-9B; the source text does not name the authors, their affiliations, a publication date, or the hardware used in the experiments.

Risks and caveats

The reported gains are benchmark results (LIBERO-Pro, robosuite, and an out-of-domain fine-tuning test on Qwen3.5-9B) rather than independently replicated or real-world deployment figures, and the source gives no detail on the robots or hardware used to obtain them.