Runta's FrontierHarness v1.1 finds the best agent harness differs by model

Runta, working with Baseten, has published FrontierHarness Eval v1.1, a benchmark of agent harnesses that the authors call the largest-scale harness leaderboard so far. It used 83B tokens across 27,280 agent runs. The two questions it set out to answer: when you switch your model, should you switch the harness too, and what is the best harness for each model?
The earlier v1.0 ranked harnesses on a single model, Kimi K3. It ran nine harnesses, and their pass rates were within 17 points of each other, but median cost per pass differed 17 times. The authors suspected this came mostly from the harness and model pair: a harness that is cheap with one model can be expensive with another. V1.1 tests that. It covers 22 harnesses x 5 models x 62 tasks x 4 attempts, 76 times the size of v1.0.
The setup. The 22 are configurations of 18 harnesses: the four DeepSeek Harness profiles have different tool sets and OpenCode ran at two versions, and each counts separately. Baseten served all five models: GLM-5.3, GLM Flash (GLM-5.3-Flash), DeepSeek (DeepSeek-V4.1-Flash-1017), Muse and Kimi-K3. The 62 tasks are terminal tasks with time limits from 15 minutes to 8 hours. The Harbor evaluation framework ran each harness with its default settings, which the authors think is closest to everyday use, with the same version and settings for every model and at maximum reasoning effort. Each trial got a fresh Runta runtime that was deleted afterwards, so no trial could see an earlier one's files. Audits of trial logs flagged 216 trials for reward hacking, 163 of them online lookups; these score 0.
The headline result. Each model had a different best-value harness: Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse and Alma with Kimi-K3. Best value means the lowest cost per pass, unless another harness costs at most 30% more per pass and passes at least 5 points more often. That exception picked ZCode over OpenCode v1 with DeepSeek (25% more cost, 7.7 more points) and Alma over Oh My Pi with Kimi-K3 (9% more cost, 6.5 more points). Cost ranks moved a lot between models: by cost per task, the median harness moved 11 of 22 places, and Codex CLI went from 1st with GLM-5.3 to 16th with Kimi-K3. Codex CLI was cheapest per pass with GLM-5.3 ($0.803) but 12th with Muse and 10th with Kimi-K3. Over all models it had the lowest cost per pass ($0.937) and Antigravity CLI the highest ($1.63).
Across all models, four harnesses sit on the cost/pass-rate frontier: OpenCode v2 (lowest cost per task, $0.561), Pi, Codex CLI and ZCode (highest pass rate, 61.9%). The rest cost up to 1.6 times as much and passed 50.6% to 60.2%.
How sure is it. The authors redrew the 62 tasks 4,000 times and picked the best-value harness each time. Codex CLI with GLM-5.3 stayed first in 99.6% of draws. The other four winners are the most likely of several close harnesses: Oh My Pi with Muse 53%, ZCode with DeepSeek 47%, Pi with GLM Flash 42% and Alma with Kimi-K3 25%.
Where the cost differences come from. A two-way analysis of variance on the log average cost per task of the 110 cells attributes 92.8% to the model (list prices differ by 20 times or more), 2.3% to a harness effect shared by all models and 4.9% to the pair, which also contains noise since each cell has one value. With the model fixed, 68% of the variation between harnesses (4.9 of 7.2 points) depends on the pair or is noise. Still, with the model fixed the most expensive harness cost 1.8 to 5.0 times the cheapest. Shuffling trials between harnesses 300 times per model gave a chance ratio of 1.2 to 1.8 times, and all five real ratios exceeded every shuffle (p = 0.003). The largest gap is with DeepSeek: Cline CLI cost 5.0 times as much as OpenCode v1, for 2.0 more points. With the other models the cheapest harness per task was at most 0.4 points behind the most expensive. Splitting attempts into two halves, cost rankings correlated at 0.71 for the same model and 0.26 across two different models, so reruns do not explain the rank moves. DeepSeek Harness: Creator moved most, from 1st with Muse to 21st with GLM Flash; OpenCode v1 was 1st with DeepSeek and 13th with Muse.
Cost does not buy score. The rank correlation between cost per task and pass rate across harnesses ran from -0.39 to +0.22 depending on the model. The highest-pass-rate harness was 5th to 20th of 22 in cost per task, and with four of five models the most expensive harness per pass passed less often than the cheapest.
Simpler does not mean cheaper. The authors compared four factors with cost per pass: start tokens, tool count, cache hit rate and steps. Only steps mattered: the rank correlation of steps per pass with cost per pass was +0.51 (p = 0.015) over the 22 harnesses, versus -0.03 to +0.02 (p of 0.88 or more) for the others. A trial used 2.0 to 4.8 million input tokens on average because each step resends the context; the largest first request, 21,365 tokens, is under 1% of that. mini-SWE-agent has the smallest first request, 882 tokens, but took 38 steps per pass and was 17th of 22 in cost per pass.
Where the money goes. At list prices, cached input was 42% of cost, output 33% and other input 25%. Cached input costs 2% (DeepSeek) to 20% (GLM Flash) of the other-input price. With DeepSeek, Cline CLI (highest cost per task) read 86% of its input from cache against 97% for OpenCode v1, and it sent twice as much input. And 64.5% of the list-price cost of scored runs went to trials that did not pass: 52.7% to failures and 11.8% to timeouts.
What the five winners share. The authors stress these are observations, not tests. One shell tool does most of the work, getting 61% to 93% of tool calls; Pi's default prompt has four tools (read, bash, edit, write) and sends file searches to bash. Requests only append to the context, so 91.1% to 94.0% of input in passed trials came from cache, though 18 of the 22 harnesses read 91% or more, so this does not set the five apart. And there are no subagents and almost no summaries: each used a subagent in at most 6 trials, and only Codex CLI summarized, in 8 trials, at 90% of the context window. They share a weakness too: about three in four failed trials ended with a confident final answer rather than the time limit (68% with Pi to 82% with Codex CLI), and few admitted the work was incomplete. Failed trials took 60% (Codex CLI) to 69% (Alma) of each harness's cost. A check before stopping could save part of that, but it was not tested.
Limitations. Only open models and one provider were used; closed models can give different results, and the cache and prices are Baseten's, so another provider can change cost ranks. The authors plan to add closed models such as Claude Opus 5.5, GPT-6 Astra and Gemini 4 Carbon.
Key facts
- FrontierHarness Eval v1.1 by Runta with Baseten: 22 harness configurations x 5 models x 62 tasks x 4 attempts, 27,280 agent runs and 83B tokens, 76 times v1.0.
- Best-value harness differs per model: Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse, Alma with Kimi-K3.
- By cost per task the median harness moved 11 of 22 places between models; Codex CLI went from 1st with GLM-5.3 to 16th with Kimi-K3.
- Only Codex CLI with GLM-5.3 is firm as best value (99.6% of 4,000 task resamples); the other four winners held in 25% to 53%.
- Steps, not prompt size, tool count or cache hit rate, track cost per pass (rank correlation +0.51); 64.5% of list-price cost went to trials that did not pass.
Why it matters
Harnesses are usually compared on one model. This benchmark tests each harness across five models and finds that cost rankings move a lot: the median harness shifted 11 of 22 places, and Codex CLI fell from 1st to 16th between GLM-5.3 and Kimi-K3. The v1.0 board had shown pass rates within 17 points but median cost per pass differing 17 times, and v1.1 tests why. The model sets the general cost level (92.8% of variance in log cost), but with the model fixed the harness still changes cost by 1.8 to 5.0 times, more than chance.
Who it affects
Anyone choosing a coding-agent harness to run an open model, and teams that switch models and keep the same harness by default. The authors' advice is that you should consider switching harnesses when switching models. It also bears on harness builders: the five best-value harnesses share a single dominant shell tool, append-only context and few subagents or summaries, though the authors call these observations rather than tests.
How to use it
The authors' reading is to treat the harness as a choice made per model, not fixed once. Their best-value pairs are Codex CLI with GLM-5.3, Pi with GLM-5.3 Flash, ZCode with DeepSeek V4.1 Flash 1017, Oh My Pi with Muse and Alma with Kimi-K3. Best value there means lowest cost per pass, unless another harness costs at most 30% more per pass and passes at least 5 points more often. If you want to cut cost, the data points at steps: the rank correlation of steps per pass with cost per pass was +0.51, while start tokens, tool count and cache hit rate showed essentially none. The results are on Baseten's models and prices, so rerun the comparison if your provider differs.
How solid is it
The benchmark is large (27,280 runs, 62 tasks, 4 attempts each) and the authors tested several of their own claims: a shuffle test (p = 0.003) shows the cost spread is not rerun noise, and split-half rankings correlated 0.71 for the same model but 0.26 across models. A 4,000-draw task resample supports Codex CLI with GLM-5.3 at 99.6%, but the other four winners held in only 25% to 53% of draws, so they are the most likely of several close harnesses. The post comes from Runta, which also supplies the runtime for each trial, and it is a vendor publication. Each harness and model cell has a single value, so the pair share of variance includes noise.
Risks and caveats
Only open models were tested, on one provider (Baseten), and the authors say closed models can give different results and another provider can change cost ranks. All costs are inference estimates at list prices, not billed amounts. Harnesses ran with default settings at maximum reasoning effort, which may differ from how a given team configures them. Audits flagged 216 trials for reward hacking, scored 0. The authors did not test whether a check before stopping would save cost, and too few trials used subagents to judge them. Closed models such as Claude Opus 5.5, GPT-6 Astra and Gemini 4 Carbon are planned, not yet included.
“you should consider switching harnesses when switching models to maximize your experience”
— Runta, FrontierHarness Eval v1.1