Ai2 introduces ToyBlocks, a controlled simulation framework for robot learning research

Robotics researchers have released ToyBlocks, an open simulation framework designed to study how robots learn manipulation skills in a controlled environment. The system generates its own training data, runs hundreds of parallel experiments on a single GPU, and lets researchers trace exactly why a model succeeds or fails - a capability that remains rare in robot learning research.

ToyBlocks centers on a robot arm and colored blocks on a tabletop. The setup is deliberately simple, but the tasks require reasoning about object interactions, occlusion, and spatial relationships. Blocks carry letters and pictures on their faces, so a robot must sometimes flip a block to see a hidden face before deciding where it belongs. The framework automatically generates demonstrations - recordings of a robot completing tasks in simulation - without human teleoperation. Researchers can vary the training mix by emphasizing specific skills like pushing or flipping, then test performance on held-out tasks.

The project comes from the Allen Institute for AI (Ai2) and represents the first in what the team calls FoundationWorlds: controlled simulated environments paired with tools for generating tasks, training demonstrations, and behavioral metrics. Each FoundationWorld targets a specific robotics domain. The narrow scope of ToyBlocks makes it feasible to satisfy four design requirements simultaneously: realistic physics and appearance, automatic generation of tasks and demonstrations at scale, challenges that combine reasoning with dexterity, and broad coverage of possible tasks within the domain.

How the data pipeline works

ToyBlocks uses reinforcement learning and motion planning to generate skills for short tasks - pushing a block across the table, flipping it onto another face without grasping. Skill training runs in Nvidia's Isaac Lab framework. The system can draw on privileged information from the simulator, such as exact block positions, to generate skills that would require teleoperation in the real world. A task-and-motion planner then sequences these skills to solve longer tasks. Running hundreds of plans in parallel on a single GPU achieves up to 24 times real-time performance.

The initial data release includes 85,336 long-horizon demonstrations, each comprising roughly 6 to 12 skill demonstrations. The datasets cover nearly 1,600 hours of robot execution data and 144 million frames. A larger release with additional skills and benchmark tasks is planned within weeks. The team is releasing simulation assets, procedural task and demonstration generators, training code, evaluation code, and two trained models based on Molmo 2 and ManiFlow.

What the experiments revealed

The researchers used ToyBlocks to test several claims about robot learning. One set of experiments examined how well vision-language-action models (VLAs) generalize spatially. Despite large-scale pretraining, popular action models generalized poorly when tested on new block configurations, stack heights, or workspace regions outside their training range. "We believe action model generalization remains far from solved," the team wrote. ToyBlocks provides a testbed for systematically isolating which techniques improve generalization.

In long-horizon tasks, the team tested hierarchical systems that pair a high-level planner with low-level execution models. Using Molmo 2 as the planner and ManiFlow as the execution policy, the system succeeded on 55.5% of simulated spelling tasks and 50.0% of real-world episodes when transferred to a physical Franka FR3 arm. The sim-to-real transfer required three stages: system identification to match real-robot dynamics, domain randomization during data generation, and depth estimation from RGB cameras at inference time.

Experiments with frontier models showed that Claude Opus 5.5, given access to ToyBlocks skills through the Model Context Protocol (MCP), solved all 30 block-stacking tasks in both text-based and vision-based closed-loop settings. In a search-and-spell scenario requiring the agent to find letter blocks and spell three-letter words, the agent succeeded or correctly declared tasks infeasible 63% of the time. Errors stemmed mostly from confusing similar-looking letters or choosing incorrect orientations.

The team also tested whether an agent could learn how its skills behave through experimentation. When an agent ran its own tests and wrote its own skill descriptions, success rates rose from 24% with default descriptions to 63%, outperforming the 36% achieved with descriptions written by a human expert.

World models as simulators

Action-conditioned world models predict how a scene changes in response to robot actions. ToyBlocks provides ground-truth simulation states, enabling controlled tests of these models. The team tested three model types on 900 trajectories per skill - half successful, half unsuccessful. Wan 2.2 predicted block motion and task outcomes most accurately. Astra tracked the gripper well but struggled with contact simulation. LeWM was fast and accurate over short segments but drifted over full trajectories. Models that scored higher on pose and success prediction also produced better training data: adding 100 model-generated trajectories from the best video-generation model raised a policy's success rate in an unseen region from 19% to 78%.

Why this matters for researchers and product developers

ToyBlocks changes the economics of robot learning research. Instead of requiring expensive hardware and hours of human teleoperation, researchers can generate thousands of demonstrations automatically and run controlled experiments that isolate specific variables - training data composition, model architecture, spatial generalization - with clear ground truth. For product teams building manipulation systems, the framework offers a way to stress-test models before deployment and understand failure modes systematically. The sim-to-real transfer results suggest that policies trained entirely in ToyBlocks can work on physical hardware, though success rates remain modest enough that the problem is far from solved. The tools are open, and the team is encouraging other labs to build their own FoundationWorlds on the same principles.