Glonce

SO-101 Bench introduces a simulated test for low-cost robot arms

researchThe Glonce editorsHugging Face🔥58Glonce rating58Impact45Novelty60Relevance95Surprise30Players12Credibility75
SO-101 Bench introduces a simulated test for low-cost robot arms

A new paper introduces SO-101 Bench, a simulated benchmark for language-conditioned spatial manipulation on the SO-101, a low-cost robot arm. The authors say such arms have become the most common entry point to robot learning, yet no released policy performs spatially constrained, language-conditioned tasks on them out of the box. They add that existing SO-101 benchmarks run only on physical hardware, with simple tasks and few held-out objects.

The benchmark is built on an Isaac Lab digital twin of a real SO-101 setup. It has four tasks scored as five families, and 138 objects divided into seen, unseen-instance and unseen-class splits. It includes 916 evaluation episodes, with subsets for held-out appearances, rotate-to-fit planning and occlusion memory. It also supplies 3,629 simulated demonstrations. Every learned policy is fine-tuned on these demonstrations under an equal compute budget.

The authors test whether the twin ranks policies the way the real robot does. They took five policies trained without real data, four VLAs and an LLM agent, and ran the same weights on the physical arm in an unseen environment with unseen objects. The ranking stayed exactly the same (Spearman rho = 1.00, MMRV = 0.000), although simulation overestimates success.

Across nine VLAs and world-action models, the best learned policies score near 61%. Spatial placement, unseen object classes and rotate-to-fit planning remain open problems for them. An LLM agent that writes code and hands grasping to a learned policy scores 83.2%. An ablation shows that this lead comes from the choice of harness and the choice of language model.

Key facts

  • SO-101 Bench is a simulated benchmark on an Isaac Lab digital twin of a real SO-101 low-cost arm setup, covering language-conditioned spatial manipulation.
  • It has four tasks scored as five families, 138 objects across seen, unseen-instance and unseen-class splits, 916 evaluation episodes and 3,629 simulated demonstrations for fine-tuning.
  • Across nine VLAs and world-action models, the best learned policies score near 61%; spatial placement, unseen object classes and rotate-to-fit planning remain open.
  • An LLM agent that writes code and hands grasping to a learned policy scores 83.2%; an ablation attributes the lead to the choice of harness and language model.
  • Five policies trained without real data keep their exact ranking on the physical arm (Spearman rho = 1.00, MMRV = 0.000), though simulation overestimates success.

Why it matters

The authors argue there is a gap: the SO-101 is the most common entry point to robot learning, but no released policy handles spatially constrained, language-conditioned tasks on it out of the box, and existing SO-101 benchmarks run only on physical hardware with simple tasks and few held-out objects. A simulated benchmark with held-out object classes and dedicated subsets for rotate-to-fit planning and occlusion memory targets exactly that gap. The headline result is a split: the best learned policies sit near 61%, while an LLM agent that writes code and delegates grasping to a learned policy scores 83.2%.

Who it affects

Anyone working with low-cost arms such as the SO-101 and wanting to compare language-conditioned policies without access to the same physical hardware. It is also relevant to teams building VLAs and world-action models, since the paper reports that spatial placement, unseen object classes and rotate-to-fit planning remain open for the best of nine such models, and to those building LLM agents that write code for robots.

How to use it

The benchmark is meant for evaluating policies: each learned policy is fine-tuned on the 3,629 simulated demonstrations under an equal compute budget, then scored on the 916 evaluation episodes, including the seen, unseen-instance and unseen-class object splits. The source text mentions no code, dataset release or license, so availability is unknown from this abstract.

How solid is it

The figures come from the paper's own abstract and are self-reported. The strongest piece of evidence is the sim-to-real check: five policies trained without real data kept the same ranking on the physical arm in an unseen environment with unseen objects, with Spearman rho = 1.00 and MMRV = 0.000. The 61% figure is stated only as "near 61%", and the source does not say whether it is a mean, a maximum or a per-model score. The names of the nine models, and the LLM and harness behind the 83.2% agent, are not given in the source text.

Risks and caveats

Simulation overestimates success relative to the real robot, and the source does not state by how much, nor does it give the real-robot success numbers. The perfect rank agreement covers five policies, four VLAs and one LLM agent, so it says the ordering held for that set, not that simulated scores match physical ones. The LLM agent's lead depends on the choice of harness and the choice of language model, according to the authors' ablation, so it is not a property of the approach alone. The names of the four tasks and five families are not listed in the source text.

“Low-cost arms such as the SO-101 have become the most common entry point to robot learning, yet no released policy performs spatially constrained, language-conditioned tasks on them out of the box”

— SO-101 Bench paper abstract