Glonce

HEELO framework builds agent harnesses from LLM-backed primitives

agentsThe Glonce editorsHugging Face🔥55Glonce rating55Impact40Novelty55Relevance95Surprise30Players20Credibility50
HEELO framework builds agent harnesses from LLM-backed primitives

The paper starts from a problem it calls the last mile to the real world. Frontier models with their own harnesses still fall short on real-world benchmarks: on Terminal-Bench 4.0, GPT-5.6 Sol with Codex resolves 37.3% of the tasks and Claude Opus 5 with Claude Code resolves 51.8%. The authors say these tasks fail at the step from reaching a resource to applying it. Existing approaches put the resource within reach but leave the competence to apply it with the caller, so each harness rebuilds that competence.

The proposed fix has two parts. Agent-native primitives are reusable units that equip a resource with a resident LLM, which applies the resource in context and communicates with other primitives. HEELO (Harness Engineering around Evolving Agent-Native Primitives for the Last Mile to Real-WOrld Tasks) is a harness engineering framework that builds each task's harness from a library of such primitives. For a given task, HEELO composes the primitives it needs into a task-specific graph and verifies the result against evidence from the environment. It revises the primitives a failure implicates, and it admits new or improved primitives only after validation.

The authors instantiate HEELO with eight primitive families, each built for one kind of task, and evaluate them on more than twenty benchmarks. The abstract names four families and their results. Dev-Primitives raise Terminal-Bench 4.0 resolution from 37.3% under Codex to 55.5% on the same backbone. Harness Primitives composed per task raise Terminal-Bench 2.0 Pass@1 from 60.0% for the seed harness to 72.2%. Tool Primitives raise the ToolBench pass rate from 55.8% to 75.1%. Navigation Primitives raise GAIA-Text-103 accuracy from 45.6% to 58.3%.

The last test uses the 46 real-world tasks of the authors' heelo.dev platform. There, Claude Opus 5.5 with Claude Code and GPT-6.1 Sol with Codex complete 9 tasks each on their own. Connected to HEELO through the authors' MCP server, they complete 39 and 35. The authors also report 1.9 times and 2.3 times less time and 1.55 times lower cost per completed task.

Key facts

  • Agent-native primitives are reusable units that pair a resource with a resident LLM that applies it in context and talks to other primitives.
  • HEELO composes a task-specific graph of primitives, verifies it against environment evidence, revises the primitives a failure implicates, and admits new ones only after validation.
  • Reported gains: Terminal-Bench 4.0 from 37.3% (Codex) to 55.5%; Terminal-Bench 2.0 Pass@1 from 60.0% to 72.2%; ToolBench from 55.8% to 75.1%; GAIA-Text-103 from 45.6% to 58.3%.
  • On the 46 tasks of the authors' heelo.dev platform, two frontier model-and-harness pairs complete 9 tasks each alone and 39 and 35 when connected to HEELO via an MCP server.
  • HEELO is instantiated with eight primitive families and evaluated on more than twenty benchmarks.

Why it matters

The paper targets a specific gap: models can reach a resource but often fail to apply it, and every harness ends up rebuilding that skill. Its answer is to package the skill with the resource as a reusable unit with its own LLM, so a harness can be assembled from parts instead of written from scratch. The baselines it cites are far from saturated: 37.3% for GPT-5.6 Sol with Codex and 51.8% for Claude Opus 5 with Claude Code on Terminal-Bench 4.0.

Who it affects

Teams building agent harnesses and tool-use pipelines are the most direct audience, since the framework is about how a harness is put together. Users of coding agents such as Codex and Claude Code are also in view: the heelo.dev test connects both to HEELO through an MCP server.

How to use it

The abstract describes one concrete route: connect a coding agent to HEELO through the authors' MCP server on the heelo.dev platform. Beyond that platform and MCP server, the abstract mentions no code, dataset or licence release. It gives no pricing or access terms.

How solid is it

All numbers come from the authors' own abstract. No independent replication or third-party verification is mentioned. The 46-task platform belongs to the authors. The time and cost figures (1.9 times and 2.3 times less time, 1.55 times lower cost) come without a stated baseline or a mapping to each model. The Dev-Primitives result is described only as running on 'the same backbone' as Codex, with no model named.

Risks and caveats

The abstract gives no limitations or failure analysis of HEELO. The eight primitive families are not listed; only Dev, Harness, Tool and Navigation Primitives are named. The abstract names no authors or institutions. Gains measured against a seed harness or a vendor harness on selected benchmarks do not by themselves show the same lift on other tasks.

“These tasks fail at the step from reaching a resource to applying it, which we call the last mile to the real world.”

— From the paper's abstract