Claude Opus 4.8, GPT-5.5 show no consistent edge from native harness

An agentic coding system pairs a language model with a harness: the tools, prompts and control flow that let it act as an autonomous software engineer rather than a chat partner. Vendors build harnesses tuned to their own models, and the common assumption is that this vendor-native pairing solves more tasks than swapping in a neutral, cross-model harness. A new paper tests that assumption directly, using a private suite of 256 repository and post-cutoff contest tasks kept out of public view, with the contest problems newer than the models' training cutoffs so the answers could not have been memorized.

The comparison is paired by model. On Claude Opus 4.8, the same 80 tasks ran once under the vendor-native Claude Agent SDK and once under deepagents, a neutral, cross-model harness; on GPT-5.5, the same 80 tasks ran once under the vendor-native OpenAI Codex SDK and once under deepagents. Gemini 3.5 Flash and DeepSeek-V3.2 were run as side cells, but the abstract reports no harness comparison or solve rate for either model. An isolated oracle graded 792 of the 800 runs the study had planned.

On Claude Opus 4.8, the native harness averaged 48.8% against 50.0% for deepagents, trailing by 1.25 percentage points, with a 95% confidence interval of -10.0 to +7.5 that spans zero. On GPT-5.5, the native harness averaged 55.6% against 54.4% for deepagents, leading by 1.25 percentage points, with a confidence interval of -4.4 to +6.9 that also spans zero. Neither gap resolves an average advantage for either harness, the authors write.

The Opus 4.8 average hides a sharp split. Broken down by task type, the native harness trailed deepagents by 9.0 percentage points on the 61 repository tasks but led it by 23.7 percentage points on the 19 contest tasks, a difference a permutation test put at about a 0.3% chance of arising at random (p = 0.003). The authors are explicit that this split was chosen after seeing the results, not decided in advance, and say it needs a designed replication before it can be relied on.

Separately, the study surfaces a measurement issue that cuts across the whole benchmark: of 81 runs cancelled after hitting a wall-clock time limit, 22 had already produced a patch that passed grading before being cut off. The abstract does not attribute these runs to a specific model or harness.

Cost tells a similar story of uncertainty. Re-priced from raw per-turn usage at frozen list prices, deepagents cost 1.3 to 1.6 times as much per solved task as the native harness on Claude Opus 4.8, and 1.2 times as much on GPT-5.5. The authors call these observed-usage estimates rather than settled billed figures: on the Anthropic account, 58 runs left no usage record at all, and depending on how that missing spend is allocated between the two harnesses, the Opus 4.8 cost ratio could land anywhere from 0.7 to 2.3. Which harness is actually cheaper to run on Opus 4.8 is, in other words, unresolved.

The paper is a revision of an August 2026 manuscript, corrected after the authors found a usage-semantics defect in their own telemetry that had distorted its earlier cost figures. Alongside the write-up, they release the orchestrator, the grading oracle, reanalysis code and derived aggregates; the 256 underlying tasks stay private.

Key facts

On Claude Opus 4.8, the vendor-native Claude Agent SDK averaged 48.8% against 50.0% for the neutral deepagents harness, a statistically inconclusive 1.25 percentage point gap (95% CI -10.0 to +7.5).
On GPT-5.5, the vendor-native OpenAI Codex SDK averaged 55.6% against 54.4% for deepagents, also inconclusive at a 1.25 percentage point gap (95% CI -4.4 to +6.9).
Opus 4.8's average hides a sharp split: the native harness trailed by 9.0 percentage points on 61 repository tasks but led by 23.7 percentage points on 19 contest tasks (p = 0.003), a pattern the authors call exploratory pending a designed replication.
Of 81 runs cancelled at a wall-clock time limit, 22 had already produced a patch that passed grading before being cut off.
Deepagents cost 1.3 to 1.6 times as much per solved task as the native harness on Opus 4.8, and 1.2 times as much on GPT-5.5, but 58 runs with no usage record leave the real Opus 4.8 cost ratio anywhere from 0.7 to 2.3.

Why it matters

Teams building coding agents generally assume that a vendor's own harness, tuned to its own model, will beat a generic one. This is a rare same-model, paired-harness test of that assumption, run on a private task suite built to resist contamination: the contest tasks are newer than the models' training cutoffs, so the answers could not have been memorized. The assumption stays unresolved on average for both models; on Opus 4.8, the harness effect is not constant: it flips sign depending on the kind of task, a pattern the authors call exploratory pending a designed replication, and one that would be more useful, and more unsettling, than a single average number would be for anyone comparing agent setups.

Who it affects

Anyone choosing or building an agentic coding setup: teams deciding between a vendor SDK (Claude Agent SDK, OpenAI Codex SDK) and a cross-model orchestration framework like deepagents, and anyone citing a vendor benchmark that assumes the native pairing wins by default. It also affects how coding-agent benchmarks get read in general. The wall-clock-ceiling finding, where already-correct patches got cancelled, and the post hoc repository/contest split both show specific ways solve-rate numbers can mislead if taken at face value. Gemini 3.5 Flash and DeepSeek-V3.2 were run as side cells but got no reported comparison in the abstract, so the evidence available here covers only Claude and GPT-5.5.

How to use it

There is no product to buy here, but there is tooling: the authors release the orchestrator, the grading oracle, reanalysis code and derived aggregates, so another team can rerun the analysis or check the grading logic. The 256 underlying tasks are not released, which is the point of a contamination-controlled suite, but it also means nobody outside the authors can reproduce the headline numbers on the identical benchmark. The practical takeaway for a team picking a harness: test both the vendor-native and a neutral option on your own task mix rather than assume either wins by default, since neither showed a resolved average edge, and on Opus 4.8, an exploratory, unreplicated split even flips sign between repository maintenance and contest-style problems.

How solid is it

This is an arXiv preprint, so it has not been through journal or conference peer review, which matters for a paper reporting on its own results. The design is nonetheless a real strength: pairing the same model against itself under two harnesses isolates the harness effect from model quality, and both headline gaps come with 95% confidence intervals that the authors report as including zero, meaning neither is statistically resolved. The repository/contest split is more fragile. It was chosen after looking at the data, and while a permutation test puts only about a 0.3% chance on it arising randomly (p = 0.003), the authors themselves call it exploratory and say it needs a designed replication. Coverage was high: 792 of 800 planned runs were graded. Working against confidence: the two side-cell models got no reported harness comparison in the abstract, and this write-up is itself a revision that corrects a real bug in the authors' own prior cost telemetry, a sign the underlying measurement is genuinely hard to get right even for the people running it.

Risks and caveats

The cost comparison is the least settled part: the ratios are estimates re-priced from usage logs at frozen list prices, not observed bills, and 58 runs on the Anthropic account left no usage record at all. Depending on how that missing spend is split between the two harnesses, the Opus 4.8 cost ratio could be anywhere from 0.7 to 2.3, so which harness is actually cheaper on Opus 4.8 is unresolved, not just imprecise. No absolute dollar figures are given in the abstract, only ratios. The 22 of 81 cancelled-but-already-passing runs were not attributed to a specific model or harness in the abstract, so it is not possible to say from the abstract whether the issue hit one cell harder than another. And because the repository/contest split was found rather than predicted in advance, it should be read as a hypothesis the paper raises, not a result it has confirmed.