Ai2 Releases ToyBlocks to Separate Robot Planning Failures From Control Failures

Ai2 released ToyBlocks on November 6, an open robot-learning simulation framework designed to help researchers determine why a robot fails. Its central contribution is experimental control: teams can change training examples, repeat tasks and replace individual components to distinguish a bad plan from poor physical execution.

The release announcement on Hugging Face describes an initial dataset of 85,336 long-horizon demonstrations, covering nearly 1,600 hours of robot execution and 144 million frames. For robotics researchers studying generalization, the useful change is the ability to generate different training mixtures and evaluate them against controlled, withheld tasks.

A small world with adjustable experiments

ToyBlocks places a robot arm above a tabletop of blocks. Letters and pictures on their faces create tasks that require searching, rotating, flipping and stacking. A hidden letter can force the robot to gather information before deciding what to move. That makes the environment useful for studying both perception and action without introducing the full variability of an open-ended household scene.

Ai2 calls this approach “FoundationWorlds”: constrained simulated environments paired with task generation, demonstrations and behavioral measurements. In its own publisher announcement, Ai2 explains that ToyBlocks combines motion planning with reinforcement-learning-based skills to generate demonstrations automatically, reducing reliance on human teleoperation. A task-and-motion planner sequences those skills into longer tasks.

The distinction between generating expert data and running a learned policy matters. Data generation can use privileged simulator information, including exact block positions and orientations. A trained policy operating from camera observations must estimate those properties. Strong demonstration generation therefore does not establish that a learned robot has solved perception or generalization.

Diagnosing which component failed

The public repository documents closed-loop, GPU-parallel evaluation on newly generated scenes or the exact initial states of recorded episodes. Task goals use PDDL, a planning description language, allowing the framework to measure success and partial progress without separate metric code for each task.

Researchers can replace either the high-level planner or the low-level controller with an oracle: a reference component supplying correct plans or actions. If a learned planner succeeds with reference execution but fails with its learned controller, that comparison helps locate the weakness. Conversely, correct high-level instructions can expose failures in manipulation. Failed states can also return to data generation to produce expert demonstrations and recoveries.

This creates a practical research loop: identify a failure, generate relevant examples, retrain and repeat the same evaluation. It connects to the broader argument in our coverage of RL environments as training infrastructure: an environment becomes especially useful when it supports repeatable experiments and informative feedback.

Physical transfer is promising, with clear limits

Ai2 reports testing policies trained entirely on simulation data on a physical Franka FR3 arm with a Robotiq gripper. ManiFlow’s individual manipulation skills achieved reported success rates of 80.0% in simulation and 81.1% in the real world.

Longer tasks were harder. The hierarchical Molmo 2–ManiFlow system completed 111 of 200 simulated spelling episodes, or 55.5%, and 15 of 30 physical episodes, or 50.0%. These evaluations used the easiest scene configuration, with block characters visible and upright. The blocks were seen during training, and operator interventions to stop unsafe motion counted as failures.

Those qualifications bound the result. Thirty physical episodes provide an initial transfer demonstration; they do not establish reliable performance across unfamiliar objects or broader environments. Ai2 itself says further trials are needed for statistical confidence across tasks. The gap between individual skills and complete spelling tasks also illustrates why component-level and end-to-end measurements both matter.

What is available now

Ai2 says the release includes simulation assets, procedural task and demonstration generators, datasets, two trained models, training code, evaluation code and validation sets. The repository documents LeRobot-format datasets and ManiFlow training infrastructure. Its main code carries an Apache 2.0 license, while submodules and downloaded assets have their own licenses.

Fully integrated reinforcement-learning training is explicitly future work, alongside a planned larger dataset and broader benchmarking. ToyBlocks’ present significance is a released framework for investigating robot-learning failures under controlled conditions. Whether those findings generalize beyond tabletop blocks remains a question for subsequent experiments.