Glonce

Senior SWE-Bench and DeepSWE coding benchmarks don't hold up to scrutiny

researchThe Glonce editorsHacker News🔥65Glonce rating65Impact60Novelty58Relevance88Surprise52Players32Credibility82

Dan Luu, who writes a long-running series of exercises on benchmarking and experimental design, devotes the middle of the seventh installment's three cases to two AI coding agent leaderboards that keep circulating online as proof one model beats another: DeepSWE and Senior SWE-Bench. Before even checking their methodology, he notes that the headline results already clash with how the models behave in practice. A surface reading of the DeepSWE homepage has OpenAI's last-generation GPT-5.5 matching Anthropic's current flagship, Claude Fable 5, while a surface reading of Senior SWE-Bench has Anthropic's last-generation Opus 4.8 beating OpenAI's current GPT-5.6. That is not what he, or anyone whose judgment he trusts, experiences using the models day to day.

On DeepSWE, the leaderboard covers 113 tasks, each run four times, with each task scored by the share of its four runs that passed (0, 25, 50, 75 or 100 percent). GPT-5.5 at its 'xhigh' effort setting scores 67 percent against Opus 4.8's 54 percent at the same setting, a gap the chart makes look as large as the one between Opus 4.8 and Gemini-3.5 Flash. Broken down by task, the two models tie on 34 of the 113, GPT-5.5 wins 57 and Opus 4.8 wins 22. Of the 79 tasks where they disagree, only four are written in Rust, the one language Luu says he actually uses coding agents for (hierarchical evaluation cancellation in Boa, deterministic multi-key sorting in fd, preserving selector structure in oxvg, and trap coredump generation in wasmi), and only one of the four even loosely resembles something he has done before; none of the four relate to what he actually uses coding agents for. By his account that leaves the DeepSWE score meaningless to him personally, and it illustrates a general problem: a single summary number can look decisive while resting on a task mix that has almost nothing to do with what a given user or team actually builds.

Senior SWE-Bench works differently, and by Luu's account worse. To count as a 'tasteful solve' there, a submission must clear a minimum rubric score and must not run to twice the length of the reference fix or longer, a hard cutoff rather than a sliding scale. On a paperless-ngx task about workflow query performance, GLM-5.2's fix passes as tasteful at 121 lines against a 61-line reference; one line longer and it would have hit the double-length cutoff and failed instead. On a Plausible task fixing blank comparison dates in a Top Pages report, the reference fix is a single line, counted as two because an addition and a deletion are each scored separately, which caps any tasteful solve on that task at three lines total. On that same task the grading also looks arbitrary at the level of individual diffs: Opus 4.8's fix passes as tasteful, Opus 4.7's fix for what Luu calls the same underlying change fails, and so does Fable 5's, which spreads the change across extra files. Luu thinks Fable 5 might reasonably fail on those grounds, but sees no good reason for Opus 4.7 to fail alongside it: he calls the Opus 4.7 diff semantically identical to the reference solution, saying the only difference is that the same one-line pipeline is spread across several lines for readability.

Luu also tested the grading itself for consistency. The published tastefulness verdicts come from a single grading pass by Sonnet 4.6. He had a coding agent rerun that same grading ten times for every GPT-5.6 Sol and Opus 4.8 condition in the benchmark: at the same model and effort level, the published verdict flipped 23 percent of the time overall (32 percent on the 'relative taste' sub-score, 5 percent on practice alignment, 3 percent on task rubric), and it differed from the typical result across those ten reruns 21 percent of the time overall (27, 3 and 2 percent on the same three sub-scores). Swapping the grading model itself, from Sonnet 4.6 to GPT-5.6 Sol, cut the number of solutions rated tasteful by more than half for both GPT-5.6 Sol and Opus 4.8.

The way results get presented compounds the problem. Someone posted a link on a Slack Luu is on, its preview snippet showing Senior SWE-Bench's headline scores as Claude Fable 5 at 29.1 percent, Claude Opus 4.8 at 25.0 percent and GPT-5.6 Sol at 24.4 percent, with someone praising it as more realistic than other benchmarks that put GPT-5.5 ahead of Opus 4.8. Luu points out that the 0.6 percentage point gap between the last two is, given the grading noise he just measured, pretty much meaningless, yet it gets presented as a real difference; most people repeating the number also drop the benchmark's own note that GPT-5.6 Sol is much cheaper than Opus 4.8. He adds that the same headline comparison tests Fable, Opus and Sonnet at 'max' effort but the GPT-5.6, GPT-5.5 and GPT-5.4 family at 'xhigh', calling the mismatch odd.

Luu says he has no expertise in AI, so before publishing he asked Aaron Levin, who ran an evals team at Anthropic for a time, to check his reasoning; Levin agreed with both the reasoning and the conclusion. Luu also cites a comment from Em Chu that community sentiment on Hacker News and Twitter is a better signal of whether a model is actually good than an LLM benchmark score, and points to Max Bitker, who runs a reinforcement learning environment startup, as someone who tracks nearly every public benchmark closely enough to predict post-release sentiment from the aggregate picture, which Luu treats as a far more time-consuming exercise than reading a single leaderboard. His overall verdict is that very few of the publicly available benchmarks look like something worth relying on to get a general sense of how good coding agents are.

This teardown is the middle case in a three-part post. The first examines a popular systems performance cheat sheet for interview prep, the sirupsen/napkin-math repository on GitHub, starred 5,400 times: its published figure of 20 nanoseconds for a random memory read should be roughly 100 nanoseconds, Luu and a friend argue, because the benchmark's loop lets the CPU overlap multiple memory accesses instead of measuring one access at a time, and its disk numbers look inflated for a similar reason, with a claimed 8 GiB/s of sequential SSD throughput running well above the 5,000 MiB/s that Google's own documentation lists as the maximum for all eight local disks attached to the cloud instance used for the measurement. The post's third case revisits the common claim, echoed in Google's AI search summaries, that all-season tires turn hard and lose grip below 7C (45F). Independent tire testing that Luu cites instead shows all-season tires beating winter tires in both dry and wet conditions above 0C (32F), by a smaller margin against 'UHP' winter tires than against the 'nordic' winter tires more commonly used in the US, which he says makes the claim that cold alone forces a switch to winter tires clearly false. Across all three cases, his point is the same: numbers and rankings that circulate widely are rarely checked against an actual measurement.

Key facts

  • On DeepSWE, GPT-5.5 at 'xhigh' effort scores 67 percent against Opus 4.8's 54 percent at the same setting, but the two tie on 34 of 113 tasks and split the other 79 by a 57-to-22 margin; of the four differing tasks written in Rust, the only language the critic says he actually uses coding agents for, just one loosely resembles work he has done before and none of the four are the kind of task he would use an agent for.
  • Senior SWE-Bench requires a fix to stay under twice the reference solution's line count to count as 'tasteful': on a paperless-ngx task, a 121-line fix passes against a 61-line reference, but one line longer would have failed it; on a Plausible task, the reference is a single line, counted as two because an addition and a deletion are each scored separately, capping any tasteful solve there at three lines total.
  • On that Plausible task, Opus 4.8's fix is graded 'tasteful' while Opus 4.7's fix for the same underlying change is graded 'untasteful' and fails; the critic calls the failing Opus 4.7 diff semantically identical to the reference solution, differing only in formatting.
  • Re-running Senior SWE-Bench's LLM grading ten times on the same model and effort level flips the published tastefulness verdict 23 percent of the time; switching the grading model from Sonnet 4.6 to GPT-5.6 Sol cuts the number of 'tasteful' verdicts by more than half for both models tested.
  • A circulating preview put Senior SWE-Bench's headline scores at Claude Fable 5 29.1 percent, Claude Opus 4.8 25.0 percent and GPT-5.6 Sol 24.4 percent; the critic calls the 0.6 percentage point gap between the last two meaningless given the measured grading noise, yet it was praised online as a more realistic ranking.

Why it matters

DeepSWE and Senior SWE-Bench are the kind of leaderboard screenshots that circulate in engineering Slack channels and on social media as shorthand for which AI coding model is better, used to justify picking one model and effort level over another for real work. This piece shows that on Senior SWE-Bench, the gap between competing models is often smaller than the noise in how the benchmark is graded: a 121-line fix passes as 'tasteful' while a hypothetical 122-line version of the same fix would fail, for no reason connected to code quality, and re-grading the same Senior SWE-Bench solutions ten times flips the published verdict 23 percent of the time. Treating either leaderboard's headline percentage as a precise measurement, rather than a rough and noisy signal, means drawing a real conclusion from a difference that may not exist.

Who it affects

Developers and teams deciding which coding agent model and effort level to use for real work, if that choice rests on a leaderboard number rather than on trying the models directly. The model vendors scored on these leaderboards: OpenAI, whose GPT-5.4 through GPT-5.6 Sol models are scored, and Anthropic, whose Opus 4.7, Opus 4.8, Claude Fable 5 and Sonnet are scored alongside them; GLM-5.2 and Gemini-3.5 Flash also appear on these same leaderboards. Of these, only the standing of GPT-5.6 Sol and Opus 4.8 on Senior SWE-Bench was shown to shift inside measured grading noise. And more broadly, anyone who treats a benchmark screenshot shared online as settled evidence rather than as one noisy data point.

How to use it

Treat any single benchmark percentage on either leaderboard as noisy rather than precise: Luu's own re-grading shows the published tastefulness verdict flips 23 percent of the time on identical inputs, and he calls the 0.6 percentage point gap between Opus 4.8 and GPT-5.6 Sol meaningless given that noise. Before trusting a score, check whether the benchmark's task mix looks like the work involved. On DeepSWE, only four of the 79 tasks where the two models disagreed were in the one language Luu actually uses agents for, and only one of those four even loosely resembles his own work; a benchmark built from a different task mix can rank the same two models the other way around. Be wary of any benchmark that turns a continuous quality score into a pass or fail with a hard length or score cutoff, since a fix one line over or under the line can flip the published verdict for reasons unrelated to code quality. And when a headline comparison mixes effort levels across models, as Senior SWE-Bench's does, check what setting each model was actually tested at before comparing the scores.

How solid is it

Luu runs a long-running series specifically on benchmarking and experimental design mistakes; this is its seventh installment. He does not present himself as an AI expert and says the reasoning applied here is the same generic benchmark scrutiny used throughout the series, not specialized domain knowledge. He backs the critique by reading the underlying task pages and code diffs rather than just the summary charts, and by re-running Senior SWE-Bench's own LLM grading himself ten times to measure how often it disagrees with itself on identical inputs. He also asked Aaron Levin, who ran an evals team at Anthropic for a time, to review the reasoning and conclusion, and Levin agreed with both.

Risks and caveats

The variance figures, the 23 percent flip rate and the rest, come from Luu's own spot check: ten re-grades run through a coding agent for every condition GPT-5.6 Sol and Opus 4.8 were tested under on Senior SWE-Bench, not an independent audit of either benchmark. The piece does not name an overall best model among GPT-5.5, GPT-5.6 Sol, Opus 4.7, Opus 4.8, Fable 5 or GLM-5.2; its point is that the published scores are not solid enough to support a ranking, not that a specific model wins. And the critique targets methodology and representativeness, not the people who built the benchmarks: it does not accuse either DeepSWE or Senior SWE-Bench of being built in bad faith.

“I usually find my eyes skipping over anything that looks like an LLM benchmark since the chances that it's worth reading are near zero.”

— Em Chu, in a comment Luu quotes