HexaShell harness lets a coding agent reach 39.7% on RoboDojo

The paper starts from an observation: agents built on vision language models (VLMs) now compete with learned policies on robot manipulation leaderboards. The authors ask how far a coding agent can go when it controls a robot instead of a computer, and where it stops.

Their system, HexaShell, keeps the harness thin. It only turns the agent's commands into robot motion. The agent drives a robot with two arms through one command line. It sees three RGB cameras and knows their calibration.

The learning loop works like debugging. After each failed attempt, the agent reads the log, finds the cause, and writes or fixes small programs called tools. What it learns is kept as code, the way a coding agent fixes a failing test. The tools may use only what the robot observes. They rely on triangulation to find where objects are, relative to each other and to the gripper. If two cameras see the same point, the point lies where the two camera rays meet. If a point lies on a known plane, such as the table or the top of a block, it lies where its ray meets that plane. When the head camera is not enough, the robot moves a wrist camera to get a second view.

The protocol is narrow. The tools are built on only ten layouts of one seed per task, then frozen and tested on ten new layouts of the same seed. The agent is Codex with GPT-6-Astra at medium reasoning effort.

On all 42 RoboDojo tasks, HexaShell reaches a success rate of 39.7% and a progress score of 48.0, averaged in the same way as the leaderboard. The authors call this a new state of the art on RoboDojo.

The breakdown shows where the approach stops. Pushing reaches 92.5% and memory tasks 65.0%, but tight insertion only 8.3%. The authors conclude that the limit is contact, not reasoning: when the agent fails, it usually sees the failure but cannot make the motion work. They open source the tools evolved so far on GitHub, under the HexaFuture organization.

Key facts

HexaShell is a thin harness: it only turns a coding agent's commands into motion for a two-arm robot, controlled through one command line, with three calibrated RGB cameras as input.
After each failed attempt the agent reads the log, finds the cause, and writes or fixes small programs called tools, keeping what it learns as code.
On all 42 RoboDojo tasks it reaches a 39.7% success rate and a 48.0 progress score, which the authors call a new state of the art.
Category results vary widely: pushing 92.5%, memory tasks 65.0%, tight insertion 8.3%. The authors say the limit is contact, not reasoning.
Tools were built on ten layouts of one seed per task, frozen, and tested on ten new layouts of the same seed. The evolved tools are open sourced on GitHub.

Why it matters

The paper tests a simple idea: treat robot control as a coding problem. Instead of training a policy, a coding agent debugs its way to better behaviour and stores the fixes as code. The authors say VLM agents already compete with learned policies on manipulation leaderboards, and HexaShell is their attempt to find how far that goes with a minimal harness. The result they report is a new state of the art on RoboDojo across all 42 tasks. Just as useful is the diagnosis: reasoning is not the bottleneck, contact is. The agent usually sees what went wrong but cannot make the motion work.

Who it affects

Robotics researchers working on manipulation and anyone comparing VLM agents with learned policies on leaderboards such as RoboDojo. The finding that contact-heavy tasks like tight insertion stay hard is relevant to anyone designing agent-based robot controllers.

How to use it

The authors open source the tools evolved so far at github.com/HexaFuture/HexaShell. The setup they describe is a coding agent (Codex with GPT-6-Astra at medium reasoning effort) that issues commands on one command line to a two-arm robot, with three calibrated RGB cameras. The tools locate objects by triangulation: two camera rays meeting at a point, or a ray meeting a known plane such as the table or a block top. A wrist camera can be moved for a second view.

How solid is it

This is a preprint, and the results are the authors' own. The 39.7% success rate and 48.0 progress score are averaged on all 42 RoboDojo tasks in the same way as the leaderboard. The tools were built on ten layouts of one seed per task and tested, frozen, on ten new layouts of the same seed. The previous state-of-the-art score on RoboDojo is not given, so the size of the improvement is unknown. No authors or institutions are named in the text, and nothing is stated about GPT-6-Astra beyond its use as the model.

Risks and caveats

Only the tools evolved so far are open sourced. Performance is uneven: tight insertion succeeds just 8.3% of the time, against 92.5% for pushing. The test layouts come from the same seed used to build the tools, so the evaluation is narrow. The text does not say which robot hardware or model is used, how many tasks fall into each category, or how many trials sit behind the headline figure beyond ten test layouts of the same seed. No cost, latency or number of agent iterations is given.

“The limit is contact, not reasoning”

— HexaShell paper