ToyBlocks introduces a controlled simulated world for robot learning

The post argues that robots are learning to fold laundry, sort parts and handle unfamiliar objects, but those demonstrations tend to be narrow and do not explain what a robot has learned or why it fails. Its analogy is the petri dish or terrarium: scientists studying a complex system start with a small, controlled environment where they can vary conditions and watch the effects. Robotics, the authors say, needs the same: researchers should be able to change the training data or model design, repeat a test, and trace the cause of a failure.

The proposal is a framework for building such environments, called FoundationWorlds. Each one pairs a controlled simulated world, aimed at a specific robotics setting, with tools that automatically generate tasks and training demonstrations, and with metrics for measuring a robot's behaviour over time. The questions it targets include how models improve as more training data becomes available, and how best to combine high-level reasoning with precise physical control.

The first FoundationWorld is ToyBlocks, an open simulation framework built around a robot arm and blocks on a tabletop. The setting is deliberately simple, but the robot still has to reason about how blocks interact and how to arrange them while pushing, flipping and stacking. Pictures and letters on the block faces allow searching, sorting and arranging tasks, and a robot may need to turn a block to see a hidden face before deciding where it belongs. Domain randomization varies block size, table height and other physical properties from scene to scene. ToyBlocks builds on earlier open robot learning work including MolmoAct 2, MolmoBot and MolmoSpaces, and the authors say they aim to extend the approach to increasingly complex environments.

ToyBlocks is designed around four requirements: realistic physics and appearance; automatic generation of tasks and demonstrations at scale; challenges that combine reasoning with dexterity; and broad coverage of the tasks possible within a domain. The authors say open-ended domains make it infeasible to meet all four, while ToyBlocks' narrow scope makes it possible.

How the data is made: skills are trained with reinforcement learning and motion planning for short-horizon tasks, in Isaac Lab, Nvidia's open-source robot-learning simulation framework. The policies learn non-prehensile skills such as pushing a block across the table or flipping it onto another face without picking it up. Skill generation can use privileged information that the simulator supplies directly but a real robot cannot, such as a block's exact position and orientation. The authors say this enables automatic generation of skills that would need teleoperation in the real world. A task-and-motion planner then selects and sequences skills to solve a task and plans the movements between them. Running each plan in simulation yields recorded demonstrations with labels for the task or skill performed. Data generation runs hundreds of plans in parallel on a single GPU, at up to 24 times real-time speed. Models are still trained on raw sensor observations, so they must estimate block poses themselves.

The data generator is controllable, so researchers can build training mixtures that emphasize particular skills or fill gaps in existing datasets. Examples given: test whether adding rotation and pick-and-place demonstrations improves flipping, or vary the balance of examples inside and outside a workspace region, always evaluating on tasks held out of training. Evaluation produces more than success rates: execution speeds, forces exerted on objects, and task progress measured with the same specifications the planning module uses. A webpage interface lets users connect Claude or another frontier model and instruct the robot in natural language.

The post lists evaluations built on ToyBlocks, covering training data and methods, spatial reasoning, generalization to new letters and block appearances, multi-step planning, transfer to physical robots, and frontier models that act through tools.

VLA spatial generalization. Vision-language-action models map language instructions and visual observations to robot actions. Prior evidence on their generalization conflicts: some studies find them brittle to changes such as object layouts and lighting, others find they generalize well. The authors argue these studies often use real-world datasets with limited spatial variation. They test four tasks (Stack, Unstack, Flip, Rotate), separating interpolation (new configurations inside the training range) from extrapolation (beyond it). Examples: train with objects spawning only in the left region and test on the right, or train on stack heights of 2 and 5 blocks and test on heights 3 and 4. Using ManiFlow, pi0.5 and MolmoAct 2, the authors say these popular action models generalize poorly in the initial results despite large-scale pretraining. They then improve generalization and analyze the contribution of each technique, including LoRA fine-tuning, and say generalization remains far from solved.

High-level planning with vision-language models. Long-horizon tasks let the authors measure sequence planning, and oracles at both the high and low levels let them isolate whether planning or execution failed. An example task is spelling "Ai2" with a block stack, where the agent must first reveal hidden letters. Molmo 2, a vision-language model, acts as high-level planner, choosing the skill and the block, while ManiFlow executes; ManiFlow is trained to pause and ask the planner for the next instruction. Three scenarios of increasing difficulty are used: letters upright and facing the camera, letters that may not be upright, and letters not visible at the start. Success rates are shown for three-letter spelling tasks, with ManiFlow or a motion planner doing the low-level work.

Agentic systems. The post studies three settings with Claude Opus 5.5. In the search-and-spell setting, ToyBlocks skills are exposed through the Model Context Protocol. The agent sees the scene through camera images, calls skills such as pick, flip or rotate by pointing at a pixel, and decides when to search, build, or declare the task complete or impossible. The text available here ends partway through this section.

Key facts

• FoundationWorlds is a proposed framework for controlled simulated robot-learning environments, each pairing a world with automatic task and demonstration generation and behaviour metrics.
• The first one, ToyBlocks, is an open simulation of a robot arm and blocks on a tabletop; it generates skills and demonstrations without teleoperation, running hundreds of plans in parallel on one GPU at up to 24 times real-time speed.
• Skills are trained with reinforcement learning and motion planning in Nvidia's Isaac Lab, using simulator-only privileged information such as exact block pose.
• In the initial results, ManiFlow, pi0.5 and MolmoAct 2 generalize poorly on four tasks (Stack, Unstack, Flip, Rotate) despite large-scale pretraining.
• Further evaluations use Molmo 2 as a high-level planner over three difficulty levels, and Claude Opus 5.5 in three agentic settings, with skills exposed through MCP in the search-and-spell setting.

Why it matters

Evidence on whether robot action models generalize is conflicting, and the authors argue that real-world datasets rarely allow clean tests because spatial variation is limited. ToyBlocks is offered as a simple, reproducible setting where training data, model design and test conditions can be changed one at a time, so a failure can be traced to planning or to execution. The metrics go beyond success rates to include execution speed, forces on objects and task progress. The authors also say they intend to extend the approach to more complex environments.

Who it affects

Mainly robot-learning researchers who train or evaluate vision-language-action models, high-level vision-language planners, or tool-using agents. The post also says the webpage interface lets people connect Claude or another frontier model and instruct the robot in natural language without robotics expertise.

How to use it

The post header links code, data, models and documentation. Researchers can generate training mixtures that stress particular skills, hold out tasks, and compare results across interpolation and extrapolation tests. They can also design their own tasks and measures, for example comparing a model that selects objects by pointing against one that takes text commands such as 'pick the yellow block'. Skill training runs in Nvidia's Isaac Lab. The text available here does not state a licence.

How solid is it

This is a project announcement by the authors, who describe their own framework and their own initial results. The available text is cut off before any quantitative results appear, and the remaining agentic experiments are also cut off, so no success rates can be reported here. No hardware specification is given for the 24 times real-time figure, and no sim-to-real transfer numbers appear even though transfer to physical robots is listed among the evaluation topics.

Risks and caveats

ToyBlocks is deliberately narrow: the authors themselves say open-ended domains cannot meet all four design requirements, and that narrowness is what makes it workable. Skill generation relies on privileged simulator information that a real robot lacks, though models are trained on raw sensor observations. The authors call their generalization findings initial, and the claim that simulated arm and blocks behave very similarly to physical ones is their own stated confidence rather than a result shown in the available text.

“We believe action model generalization remains far from solved; ToyBlocks provides a valuable testbed for systematically exploring generalization techniques.”

— ToyBlocks blog post, Hugging Face