RoboColosseum tests five VLA policies and three AI agents on real robots at 124 sites

General-purpose robot policies have to work across different robots, tasks and environments. That same variation is what makes physical trials informative, and it is also what makes them hard to reproduce. A new paper, listed on Hugging Face Papers, proposes RoboColosseum as an answer: a distributed evaluation framework that combines unified policy serving and standardized protocols across a network of locally operated robots in diverse environments.

The paper's initial study evaluates five vision-language-action (VLA) policies and three frontier AI agents across four embodiments: SO-101, Franka, Bimanual-YAM and the G1 Humanoid. It runs two tracks. The Open Track compares models without task-specific adaptation, using blinded, pairwise trials under matched local conditions. The Fine-tune Track measures how well VLA policies adapt to 20 tasks across all four embodiments when each policy receives the same 100 demonstrations per task.

The headline result comes from 5,699 pairwise Open Track comparisons at 124 sites. Frontier agents rival specialized VLA policies, while all systems struggle to complete open-world tasks reliably. After fine-tuning, GR00T N1.7 leads on SO-101, and pi0.5 leads on Franka and Bimanual-YAM.

The two tracks do not tell the same story. Policy rankings show weak agreement across embodiment-specific task suites, with a Kendall W of 0.24. Open Track rankings also poorly predict relative performance after fine-tuning, with substantial rank reversals on all three embodiments shared between the tracks.

The authors also argue for measuring partial progress, which reveals differences that task success alone misses. The G1 Humanoid is the example: no policy completes any of its five tasks, yet average progress ranges from 13.3% to 23.2% across policies.

On efficiency, Open Track rankings stabilize as trials accumulate, and active matchmaking achieves comparable ranking agreement with substantially fewer physical trials than uniform sampling. The paper concludes that distributed physical evaluation can show where robot policies generalize, how well they adapt to new tasks, and how far they get on tasks they fail to complete.

Key facts

RoboColosseum is a distributed evaluation framework with unified policy serving and standardized protocols across a network of locally operated robots in diverse environments.
The first study covers five VLA policies and three frontier AI agents on four embodiments (SO-101, Franka, Bimanual-YAM, G1 Humanoid), with 5,699 pairwise Open Track comparisons at 124 sites.
Frontier agents rival specialized VLA policies in the Open Track, yet all systems struggle to complete open-world tasks reliably.
After fine-tuning on 100 demonstrations per task across 20 tasks, GR00T N1.7 leads on SO-101 and pi0.5 leads on Franka and Bimanual-YAM; Open Track rankings poorly predict these results.
On the G1 Humanoid no policy completes any of the five tasks, but average progress ranges from 13.3% to 23.2%, which task success alone would hide.

Why it matters

Physical robot trials are hard to reproduce because every lab has its own robot, tasks and surroundings. RoboColosseum tries to turn that variation into a strength by running blinded, pairwise comparisons at many sites under matched local conditions, rather than at a single lab. The scale reported is 5,699 Open Track comparisons at 124 sites. The findings also challenge a common shortcut: Open Track rankings poorly predict relative performance after fine-tuning, with substantial rank reversals on all three embodiments shared between tracks. A model that looks best out of the box is not necessarily the best one after adaptation.

Who it affects

Robotics researchers and teams building or choosing VLA policies are the main audience, since the study compares such policies and shows how rankings shift between zero-adaptation and fine-tuned settings. Groups that run frontier AI agents on robots are affected too, because the Open Track puts three of them against specialized VLA policies and finds the agents rival them. Anyone who designs robot benchmarks has a stake in the findings on partial progress and on active matchmaking.

How to use it

The abstract describes the framework's components: unified policy serving, standardized protocols and a network of locally operated robots. It also describes two evaluation modes. Use the Open Track idea to compare models without task-specific adaptation, and the Fine-tune Track idea to compare how policies adapt when each gets the same 100 demonstrations per task. For practical evaluation design, the paper's takeaways are to report partial progress alongside task success, and to consider active matchmaking, which the authors say reaches comparable ranking agreement with substantially fewer physical trials than uniform sampling.

How solid is it

This is a preprint, and the retelling rests on its abstract. The evidence base is large for physical robotics: 5,699 pairwise comparisons at 124 sites, plus a fine-tuning study of 20 tasks with matched demonstration counts. Trials are blinded and pairwise under matched local conditions, which supports fair comparison. The conclusions are the paper's own claims, not independent verification. The abstract names no authors or institutions.

Risks and caveats

Several details are not given in the abstract. It does not name the three frontier AI agents or the other three VLA policies beyond GR00T N1.7 and pi0.5. It gives no absolute success rates for any policy on either track, other than the G1 Humanoid progress range of 13.3% to 23.2%. It names no leader for the G1 Humanoid, says only that no policy completes any of its five tasks, and puts no number on the saving from active matchmaking. It does not say which three embodiments are shared between the tracks. The weak agreement across embodiment-specific task suites (Kendall W = 0.24) also means a ranking on one embodiment should not be assumed to hold on another. Even the best systems struggle to complete open-world tasks reliably.