Glonce

HarnessTax study: coding agent harness swings cost, not accuracy

agentsThe Glonce editorsHacker News🔥71Glonce rating71Impact65Novelty72Relevance92Surprise58Players35Credibility78

Researchers from UC Berkeley's Sky Lab and Arena, led by Melissa Z. Pan with co-authors Shuo Yang, Negar Arabzadeh, Wei-Lin Chiang, Ion Stoica and Matei Zaharia, published a study called HarnessTax that isolates the effect of a coding agent's harness, the software layer that manages a model's tools, context and task execution, from the effect of the underlying model. The team ran 21 model-harness pairs spanning seven models and three harnesses, Claude Code, Codex CLI and Pi, on the same 30 randomly sampled tasks from each of two benchmarks, SWE-bench Lite and Terminal-Bench 2.0. Each pair ran three times per task, each attempt was capped at 100 agent turns, and costs were computed from a fixed direct-API price list dated September 1, 2026, applied identically across harnesses. Success rates were estimated with 95% confidence intervals from 10,000 bootstrap resamples.

The headline finding is that harness choice affects cost far more than it affects whether a task gets solved. The average harness effect on success rate stayed within about plus-or-minus 2 percent on SWE-bench Lite and plus-or-minus 5 percent on Terminal-Bench 2.0. Claude Fable 5 illustrates the gap directly: it solved 97.8% of attempts in Claude Code versus 96.7% in Codex and 96.7% in Pi, a difference of about one percentage point, yet Claude Code cost about twice as much per attempt as Pi ($1.33 versus $0.67). Averaged across shared models with geometric means, Claude Code cost about 2.0 times as much as Pi and 1.6 times as much as Codex on SWE-bench Lite, and about 1.5 times as much as Pi on Terminal-Bench 2.0. The authors call this a hidden 'harness tax' that a team pays by accepting a coding agent's default harness without comparing alternatives.

The second finding is that a minimal harness can keep up with richer ones. Pi provides only four tools, read, write, edit and bash, yet it reached the Pareto frontier (the best success rate available at a given cost) on both benchmarks. Turn counts help explain where the extra spending in richer harnesses goes: for Claude Fable 5 on SWE-bench Lite, Pi and Claude Code averaged nearly identical turns per attempt (15.4 versus 15.3), so Claude Code's roughly double cost bought only a 1.1 percentage point higher success rate. Measuring the first model call in each attempt, the study found Claude Code's mean initial context was more than 10 times larger than Pi's across all seven models tested, driven by longer instructions and larger tool schemas.

The third finding is that a model does not necessarily do best in its own provider's harness. Providers sometimes tune models for their own tooling. OpenAI, for instance, describes GPT-5-Codex as optimized for agentic software engineering inside Codex. But across six Anthropic and OpenAI models and both benchmarks, an alternative harness produced the highest observed success rate in nine of twelve comparisons. Sonnet 4.6 solved 68.9% of attempts in Codex versus 66.7% in Claude Code on SWE-bench Lite at a similar cost, and GPT-5.6 Sol reached an 83.3% success rate in Pi versus 78.9% in Codex on Terminal-Bench 2.0, at roughly half the cost ($0.42 versus $0.76).

The authors flag their own limits: the findings come from only two open-source benchmarks that the tested models may have encountered during training, and results could differ on other benchmarks and real-world workloads. They plan to publicly release their profiling traces and argue that general-purpose coding agents should prioritize cost efficiency and reliability, since many everyday tasks may not need a harness's extra features, while harder problems at the edge of a model's ability may still benefit from richer, more structured harnesses.

Key facts

  • The study tested 21 model-harness pairs (seven models, three harnesses: Claude Code, Codex CLI, Pi) on SWE-bench Lite and Terminal-Bench 2.0, three runs per task, capped at 100 turns per attempt.
  • Average harness effect on success rate stayed within about plus-or-minus 2 percent on SWE-bench Lite and plus-or-minus 5 percent on Terminal-Bench 2.0, while cost varied by up to 5x for similar success rates.
  • Claude Fable 5 solved 97.8% of attempts in Claude Code versus 96.7% in both Codex and Pi, but Claude Code cost about twice as much per attempt as Pi ($1.33 versus $0.67).
  • Pi, a minimal open-source harness with only four tools (read, write, edit, bash), reached the Pareto frontier on both benchmarks, and Claude Code's mean initial context ran over 10x larger than Pi's.
  • Across six Anthropic and OpenAI models on both benchmarks, an alternative harness beat the model's own provider harness in nine of twelve comparisons, including Sonnet 4.6 doing better in Codex than in Claude Code.

Why it matters

Most coding-agent evaluations compare models while treating the harness as an afterthought, or use whatever harness ships by default. HarnessTax argues that choice quietly costs money without buying accuracy: on the two benchmarks tested, a team could pay up to 5x more per task and see essentially the same success rate. The study names this a 'harness tax' and asks evaluations to report cost and success across multiple harnesses for the same model rather than one harness alone.

Who it affects

Anyone choosing between coding-agent products (Claude Code, Codex CLI, Pi and similar tools) for day-to-day development work, teams running large volumes of agent tasks where per-attempt cost compounds, harness builders deciding how much scaffolding to add, and researchers who benchmark coding agents and need to control for harness effects rather than attribute everything to the model.

How to use it

The study's practical takeaway is to benchmark cost and success rate across more than one harness before committing to a coding agent's default setup, since a leaner harness like Pi reached the same accuracy frontier at lower cost. The authors say they will publicly release their profiling traces, which would let others repeat or extend this cost-versus-success comparison on their own workloads.

How solid is it

The setup is reasonably rigorous for a blog-style study: 21 model-harness pairs, three runs per task on 30 randomly sampled tasks per benchmark, a fixed price list applied identically across harnesses, and 95% confidence intervals from 10,000 bootstrap resamples. The authors themselves flag the main limit: SWE-bench Lite and Terminal-Bench 2.0 are open-source benchmarks the tested models may have seen during training, so the findings are explicitly scoped to those two benchmarks rather than claimed as general.

Risks and caveats

The authors state plainly that results may differ on other benchmarks and real-world workloads, and they do not claim the pattern generalizes beyond the two benchmarks tested. Providers do optimize some models for their own coding environments (OpenAI cites this for GPT-5-Codex in Codex), so a provider's own harness could still matter more on harder tasks outside this study's scope. The piece is a research blog post with an informal tone in places, not a peer-reviewed paper, and it does not disclose a total budget for the study, only per-attempt costs.

“Harness choice has little effect on task success rate, but can significantly affect the cost on the benchmarks we test.”

— HarnessTax study authors