SlopCodeBench update: GPT-6 Astra leads at 16.3%, Opus 5.5 does better on medium effort

Dex Horthy, with contributions from Gabe Orlanski, published Part I of an October 2026 SlopCodeBench writeup on October 15, 2026. SlopCodeBench gives a coding agent a problem in stages: it writes a program, receives new requirements, and has to change its existing code. Later checkpoints test both new and old behavior. In May, the last time the benchmark was published, gpt-5.5 had a 14.8% strict pass rate. This month the author ran the frontier models through the same campaigns, and the top scorer was gpt-6 astra at 16.3%.

The headline metric is the strict pass rate: the share of checkpoints where the model passes every test category, including Regression. The tested models were Opus 5.5, Fable 5.1, SOL 6 and 6.1, Astra and GLM 5.3, and their strict pass rates fell in a 12.2-16.3% range. The looser core pass rate, where all tests in a nonempty Core category must pass, landed at 65.8-74.0%. This time the author ran the full set of 36 problems, 196 checkpoints per setting, so the earlier three-problem and six-problem subset runs cannot be used to measure improvement or decline. That includes the earlier 30% Fable 5 score, which covered only 6 of 36 problems. Each model and effort setting was run once, using the pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case runner pass policy, and Docker with Python 3.12 and uv.

The finding the author digs into is effort. For some newer models, medium effort did better than XHigh. XHigh costs 2.11-4.22x as much as Medium in the three complete pairs, and Opus 5.5 did better on medium than on xhigh. Strict passes across settings ran from 24 to 32 of 196, and core passes (including strict) from 129 to 145. Astra XHigh has the most strict passes, but its lead over SOL 6.1 Medium is only two checkpoints. Opus Medium leads on core success at 145/196 but passes only 28/196 strictly. XHigh costs more in all three pairs, and strict success rises only for Astra. Recorded cost estimates span $178.51 to $1,550.07 within a 12.24-16.33% strict-score range. GPT-6.1 Medium has the lowest recorded cost at $178.51, with the same strict count as Astra Medium; Fable's recorded estimate is about 8.68x as much, with three fewer strict passes.

The author is explicit about the limits: this does not prove that more reasoning hurts coding, only that higher effort was not a reliable way to buy more strict passes in these runs. One campaign per setting cannot show how much is run-to-run variation.

The second half looks at the code the agents leave behind. Only each problem's last checkpoint counts. The author's first instinct was to drop scratch and temp files from the quality scores, but Gabe argued the benchmark should measure how much garbage models leave in the codebase, so the main results analyze all leftover code (apart from a fixed list of folders for installed packages, virtual environments and caches). For comparison, a filtered score removes folders named _work, tmp, temp, scratch, generated, build or dist, plus *_pb2.py and *_pb2_grpc.py files. In these runs the filter removed only _work files: 45 files and 41,265 source lines, all from Opus Medium's final dynamic_buffer code. Opus Medium's total drops from 216,881 to 175,616 source lines and its slop-presence to 92.90%, and it still ranks most frequent on slop presence. The filter changes nothing for the other models.

The slop figures cover the same 28 problems for all ten completed runs, because the scanner could not fully measure some files in eight problems, so those were left out for every model. Slop presence is the share of source lines flagged by at least one check, with each line counted once; flagged and source lines are totalled across the 28 problems, so larger programs weigh more (earlier posts averaged per-problem percentages instead). Opus XHigh has the lowest slop percentage at 77.75%, against 94.25% for Opus Medium, even though XHigh has fewer strict passes, 25 versus 28. The author describes the XHigh code as close to 5x cleaner. He floats a possible reason for the high numbers overall: the paper's battery of 200+ linters was built from agent-authored PRs from the months before its March 2026 publication and the v1.1 release in May 2026, and newer models may produce maintainability anti-patterns those linters do not cover. That is stated as a possibility. Fable has the lowest mean complexity at 3.57 and GPT-6 SOL Medium the highest at 11.70, with functions and methods each getting one vote. The author cautions that splitting code into more helpers can lower average complexity without making it easier to change.

His conclusion: the full benchmark is hard in a way that matters to him, because agents get the main behavior right much more often than they keep all tested behavior working as requirements change. Models are getting better at maintaining a codebase over time (14.8% to 16.3% from May to September), but not as fast as anyone would have you believe. Part II, a more qualitative look at slop patterns from Gabe and the SlopCodeBench team, is still to come.

Key facts

GPT-6 Astra scored the highest strict pass rate at 16.3%, up from gpt-5.5's 14.8% in May; the tested models ranged from 12.2% to 16.3% on 36 problems and 196 checkpoints per setting.
XHigh effort cost 2.11-4.22x as much as Medium in the three complete pairs, and strict success rose only for Astra; Opus 5.5 did better on medium than on xhigh.
Astra XHigh leads in strict passes, but only by two checkpoints over SOL 6.1 Medium; recorded costs run from $178.51 (GPT-6.1 Medium) to $1,550.07.
On code quality across 28 problems, Opus XHigh flagged 77.75% of lines as slop against 94.25% for Opus Medium, despite 25 strict passes versus 28.
The author says each setting was run once and the results do not prove that more reasoning hurts coding.

Why it matters

Most coding benchmarks test a single task. SlopCodeBench feeds an agent new requirements in stages and checks both new and old behavior, which the author says maps more closely onto how software is really built. The headline here is modest: strict pass rate rose from 14.8% (gpt-5.5, May) to 16.3% (gpt-6 astra), about 1.5 percentage points. The author reads this as steady but slower progress than claimed elsewhere, and notes that agents get main behavior right far more often than they keep everything working as requirements change. Core pass rates of 65.8-74.0% against strict rates of 12.2-16.3% show the size of that gap.

Who it affects

Teams choosing a coding model and a reasoning-effort setting are the most direct audience. In this run, paying 2.11-4.22x more for XHigh over Medium did not reliably buy more strict passes, and recorded costs ranged from $178.51 to $1,550.07 for strict scores within 12.24-16.33%. It also concerns anyone building coding-quality linters or benchmarks: the author suggests newer models may produce maintainability anti-patterns that the existing 200+ linters do not yet catch.

How to use it

The author describes how to reproduce the runs: the pinned v1.0 catalog, the stock just-solve prompt, seed 42, the any-case runner pass policy, and Docker with Python 3.12 and uv. For readers picking a setting, the practical reading is the cost-versus-strict-pass comparison: GPT-6.1 Medium had the lowest recorded cost at $178.51 with the same strict count as Astra Medium, while Fable's recorded estimate was about 8.68x as much with three fewer strict passes. Part II from Gabe Orlanski and the SlopCodeBench team will add a qualitative look at slop patterns.

How solid is it

This is one author's blog writeup with contributions from Gabe Orlanski, not a lab release. The full 36-problem run lets the results be compared with the official research findings, but each model and effort setting was run once, and the author says one campaign per setting cannot show how much is run-to-run variation. Earlier three-problem and six-problem subset runs are not comparable. The quality analysis covers only 28 of the 36 problems, because the scanner could not fully measure some files in eight problems. The source does not explain how the 'close to ~5x cleaner' figure for Opus XHigh was derived; the stated slop percentages (77.75% and 94.25%) are the figures given.

Risks and caveats

The title's 'xhigh considered harmful' framing is a provocation, not the author's conclusion: he says the results do not prove that more reasoning hurts coding, only that higher effort was not a reliable way to buy more strict passes, and that these single runs do not isolate effort as the cause. The 1.5% in the original title is about 1.5 percentage points, not a relative gain. Opus Medium's slop figure includes 41,265 lines of scratch code from _work folders; filtering them lowers it to 92.90% but it still ranks worst. Slop percentages weigh larger programs more heavily than the earlier posts did, and fewer helper functions or lower complexity do not necessarily mean easier-to-change code.

“This doesn't prove that more reasoning hurts coding. It says higher effort was not a reliable way to buy more strict passes in these runs.”

— Dex Horthy, SlopCodeBench update, October 15, 2026