Glonce

Coding-agent harnesses like Claude Code add cost, not accuracy, study finds

researchThe Glonce editorsThe Decoder🔥64Glonce rating64Impact50Novelty60Relevance95Surprise55Players25Credibility75
Coding-agent harnesses like Claude Code add cost, not accuracy, study finds

A coding agent is a language model plus a harness: the system prompt, tools and context management that let the model work inside a code repository. Makers regularly advertise score gains from new harnesses, and leaderboards mix them freely. Two researchers at Shandong University, Liu and Han, tested how much the harness matters when the model is held constant. Their paper is titled "What does a harness buy? Mostly tokens."

The authors say earlier studies report gaps of more than 20 points between harnesses, but almost always from a single run. Pure reruns of the same setup already vary by up to six points on SWE-bench Verified. Zhipu AI, for example, advertises its own harness with a 2.4-point lead over Claude Code, a margin well inside that noise range. Harnesses are also often wired into the benchmark differently, which makes comparisons less reliable still.

Liu and Han ran five models through three harnesses in their shipped versions: Claude Code, OpenCode and mini-SWE-agent. mini-SWE-agent gives the model only a Bash tool and no context management. Claude Code and OpenCode offer specialized file and search tools, long system prompts and automatic history compression. The models were Qwen3.6, Qwen3.8, Deepseek-V4-Flash, GLM-5.3-Flash and HY4-Preview; Claude Opus 5 ran as a reference only in Claude Code. Tests used the 45 hardest SWE-bench Verified tasks plus a pool of 447 more, and several setups ran twice to measure noise.

All runs were offline. In early tests with web access, 71 percent of Deepseek and Qwen model attempts pulled the correct fix straight from GitHub instead of solving the task, while Claude models almost never did. The researchers discarded those results and do not claim any maker trained on the test data, but they cut off web access for all remaining runs.

On the 447-task pool, Claude Code and mini-SWE-agent, the heaviest and lightest harnesses, sit less than two points apart for both tested models; within a five-point band they are statistically equivalent. On the 45 hardest tasks, harnesses differ by two to five solved tasks per model, while a simple rerun shifts the count by up to three. Sometimes mini-SWE-agent leads, sometimes Claude Code. The median outcome changes on 13 percent of tasks between two reruns and also on 13 percent between two harnesses; swapping models pushes it to 22 percent. Which tasks a harness wins changes from run to run, like a coin flip. The authors call the harness gap a "fresh draw of the same noise." Sample size limits detection too: with 45 tasks, even a 13-point gap shows up only half the time, and with 447 tasks gaps of about five points become reliably detectable. Zhipu's 2.4-point claim falls below both thresholds, as do many other advertised improvements.

The one measurable harness effect is a steady deficit for OpenCode, which trails the other two by up to nine points on the large pool. About half of that gap traces to an output limit: when a model exceeds it mid-step during reasoning, Claude Code prompts the model to keep going, while OpenCode lets the step end and the attempt usually fails. Adding a Claude Code-style continuation prompt to OpenCode did not help, because the same task rarely triggers the same overflow on a retry. For the rest of the gap the authors have no explanation, only an observation: under OpenCode, models stop working much sooner. On locally run models OpenCode also had a much tighter time limit than Claude Code, which the authors flag as an uncontrolled variable. Other failures: mini-SWE-agent runs into the context window because it lacks context management, mostly on tasks the other harnesses also fail, and OpenCode's search tools try to download a binary file on first use and break when offline.

Cost is where harnesses differ most. With the same model, tasks and price list, one task can cost up to three times more depending on the harness. Tool outputs barely explain this, since all three return similar amounts of text per call. The cause is the system prompt plus tool descriptions, resent with every step: about 16,600 tokens for Claude Code, roughly 7,000 for OpenCode and under 1,000 for mini-SWE-agent. Claude Code also keeps the model's reasoning output in the conversation history, based on the researchers' reverse engineering. After the first call, context grows at similar rates in all harnesses, so prompt size sets the baseline and the number of steps multiplies it. Which factor dominates depends on the model: with GLM-5.3-Flash, Claude Code costs nearly twice as much as mini-SWE-agent, while with Deepseek-V4-Flash mini-SWE-agent needs so many extra steps that the two end up even. Cheaper cached tokens dampen the gaps but do not change the ranking.

There is no home-field advantage: Deepseek's own harness scores no better with its own model than the others do, but costs the most. Claude Opus 5 delivered identical results even after the researchers removed tools from Claude Code. A year-old Claude Code version solves a similar number of tasks at less than half the cost, because Extended Thinking was not on by default back then.

Thinking was the only case where harness choice really changed the outcome. Without thinking, mini-SWE-agent with Qwen3.6 drops eleven points while Claude Code and OpenCode stay within noise, and HY4-Preview also loses eleven points in Claude Code. How much an agent depends on reasoning seems tied to the specific model and harness pairing, and the authors admit their test grid is too small to explain why. The study relies mainly on one benchmark; a small cross-check on Terminal-Bench 2.1 shows the same pattern. The authors conclude that a harness buys "fewer ways to lose a task, and a bill, not pass rate," and recommend publishing a rerun of the same setup alongside every harness comparison. Data and analysis scripts are on GitHub.

Key facts

  • On 447 tasks, Claude Code and mini-SWE-agent sit less than two points apart for both tested models; within a five-point band they are statistically equivalent.
  • Pure reruns of the same setup vary by up to six points on SWE-bench Verified, so Zhipu AI's advertised 2.4-point lead over Claude Code falls within noise.
  • The median outcome flips on 13 percent of tasks between reruns and 13 percent between harnesses; swapping models flips 22 percent.
  • Cost per task can differ up to threefold by harness, driven by resent system prompt and tool descriptions: about 16,600 tokens for Claude Code, roughly 7,000 for OpenCode, under 1,000 for mini-SWE-agent.
  • OpenCode is the one real outlier, trailing by up to nine points; about half of that gap traces to an output limit.

Why it matters

Harness gains are a common selling point for coding agents, and published comparisons often rest on a single run. This study argues that such gaps are mostly noise: a rerun of the same setup moves results about as much as swapping Claude Code for mini-SWE-agent. What does differ reliably is the bill, up to three times per task. The authors say a harness buys fewer ways to lose a task and a bill, not pass rate. The source also points to other studies in which extra effort in the harness rarely pays off, and notes that Nvidia recently cut nearly half its token costs for coding agents through better agent steering.

Who it affects

Teams choosing between coding-agent harnesses, anyone reading leaderboards that mix harnesses and models, and harness makers such as Zhipu AI whose advertised margins sit below the detection thresholds. It also touches people who run these agents at volume, since token cost per task varies far more than accuracy.

How to use it

When comparing harnesses, run the same setup twice and report the rerun alongside the comparison, as the authors recommend. Treat gaps of a few points on small task sets as inconclusive: with 45 tasks even a 13-point gap shows up only half the time, and about five points is the reliably detectable gap with 447 tasks. For cost, look at system prompt and tool description size and the number of steps the model takes. Keep thinking enabled where the model depends on it. The data and analysis scripts are on GitHub, per the article.

How solid is it

The design includes reruns, two task pools (45 hardest and 447 more) and five models, and the findings are consistent across them. A small cross-check on Terminal-Bench 2.1 shows the same pattern. Still, the study relies mainly on one benchmark, the closed-source top model, Claude Opus 5, ran only in Claude Code, and the authors admit the grid is too small to explain the thinking effects. The article does not specify which two models were used in the 447-task Claude Code versus mini-SWE-agent comparison, and it gives no absolute pass rates or dollar costs per task. This is a report on the paper by The Decoder.

Risks and caveats

OpenCode's deficit is only partly explained: about half traces to an output limit, and for the rest the authors have no explanation. OpenCode also had a much tighter time limit than Claude Code on locally run models, which the authors flag as an uncontrolled variable. All runs were offline after early web-access tests showed 71 percent of Deepseek and Qwen attempts pulling the fix from GitHub; the researchers discarded those results and do not claim any maker trained on the test data. Results may not carry over to online use. Thinking is an exception to the general finding: without it, mini-SWE-agent with Qwen3.6 and Claude Code with HY4-Preview each lose eleven points.

“What does a harness buy? Mostly tokens.”

— Title of the paper by Liu and Han, Shandong University