Codex agents barely benefit from naming a testing technique

An independent researcher ran an eval to see whether telling a coding agent, Codex running GPT-5.6 Sol, to use a named testing or verification technique improves the correctness of the code it writes, compared with giving it no extra instructions. The main task was reusing an existing benchmark: implementing Zstd in Rust. The eval covered 26 prompt conditions, among them ACL2, Alloy, "Audit and fuzz risky areas", "Audit first", Creusot, a plain Default condition with no addendum, differential testing, fuzzing, Hegel, Insta, a "Judgement" condition where the agent picks its own best technique, Kani, Lean 4, "Make no mistakes", metamorphic testing, mutation testing, property-based testing, Proptest, QuickCheck, rstest, Rust's built-in test framework, SMT solvers with Z3, cvc5 and Yices available, Spin, TDD, TLA+, and Verus. Four skills were tested on top of that: the official Hegel skill, the ECC Rust test skill (part of a skills collection with 250,000 GitHub stars and 38,000 forks), the Trail of Bits property test skill, and a skill the author wrote. Each Zstd condition and effort level (medium and xhigh) averaged 80 runs, scored on the fraction that passed 100% of a hidden test suite; a similar comparison ran on an IMAP RFC implementation task at 40 runs per condition, plus a few individual runs on other RFCs.

Before running the eval the author pre-registered predictions: TDD would underperform (55% confidence), formal methods would not overperform (52%), "Make no mistakes" would not beat no instructions at all (95%), and the ECC skill, Hegel's skill and the Trail of Bits skill would each fail to outperform (65%, 65% and 55% confidence respectively).

The results matched most of those guesses. Nothing in the 26 conditions wildly outperformed the rest. The Default condition, with no added instructions, did well above average. At xhigh effort, fuzzing and property-based-testing conditions did a little better than formal-methods conditions on average, though the picture was a lot more mixed at medium effort. The testing skills that Codex itself recommended when asked to find relevant skills underperformed, while the author's own custom skill did okay; the author attributes the difference to that skill being written to nudge agents away from their default bad habits rather than reading like a tutorial, the way the Hegel skill does. TDD underperformed as predicted, and the one skill that pushed agents toward TDD fared poorly in the runs where the agent actually followed that instruction.

Looking at what the agents actually did explains the pattern: when given the name of a technique, they mostly either wrote the same tests they would have written anyway inside a differently labeled framework, or applied the technique superficially without capturing the value it is supposed to provide. With formal methods, agents tended to prove properties that were largely irrelevant to correctness. With property-based testing, they leaned on largely random inputs, frequently hit invalid or rejected cases, or checked a trivial property against low-value random cases. The author quotes a comment from Gary Bernhardt describing agents' general approach to testing as taking pathological objections to mocks from 15 years ago and making them the backbone of the testing strategy, and says this pattern held regardless of whether the task was a bit-manipulation problem like Zstd, a protocol like IMAP, or other RFCs.

At xhigh effort, agents were generally able to get their own tests passing but the tests themselves were weak: one example given is a test meant to exercise four distinct bitstreams that instead submitted four identical bitstreams, missing a bug that only shows up when bitstreams get transposed. Running at a lower effort level in a naive loop made results worse still, with agents doing more of this and correctness stalling.

The author closes by asking why AI labs have not built reinforcement-learning environments specifically to teach agents to test well, noting that agents have gotten notably good at bounded runtime-optimization problems, a category that is cheap to generate RL environments for. Possible explanations floated are that effective testing is a harder skill to package into an RL environment than it looks, or that knowledge of effective test techniques is not widespread enough for anyone to have tried; the author does not resolve which, if either, is correct.

Key facts

26 prompt conditions plus 4 skills were tested on Codex running GPT-5.6 Sol, building a Rust Zstd implementation, averaging 80 runs per condition and effort level (medium and xhigh).
Default, meaning no added instructions at all, did well above average; none of the 26 conditions wildly outperformed the field.
At xhigh effort, fuzzing and property-based testing did a little better than formal-methods conditions on average; the picture was a lot more mixed at medium effort.
TDD underperformed as predicted, and the tested skills Codex itself recommended (including a 250,000-star, 38,000-fork ECC skill and the Hegel skill) also failed to outperform, while the author's own custom skill did okay.
Even at xhigh effort where agents got their own tests passing, the tests were often weak, for example submitting four identical bitstreams to a test meant to exercise four distinct ones and missing a transposition bug.

Why it matters

The eval was built to test a specific, common piece of prompting advice: that telling a coding agent to use a named testing technique or library will make its code more correct. The author frames it against a broader worry that it is easier than ever for agents to hit some quality bar while software quality built with them seems to be getting worse, suggesting the default habits developers rely on are not doing the job. The finding that naming a technique mostly does not help, and that plain default behavior often does about as well or better, cuts against a lot of casual prompting advice circulating for agentic coding.

Who it affects

Anyone prompting coding agents with testing instructions on the strength of having heard a technique or library is good practice, without testing expertise of their own to judge whether the agent is actually applying it well. It also affects authors of testing skills meant to be attached to agents: the skills Codex surfaced as top recommendations when asked to find relevant ones, including a widely starred ECC skill and the official Hegel skill, underperformed in this eval.

How to use it

The practical takeaway is not to assume that appending "use TDD" or "use formal methods" or "use QuickCheck" to a prompt will improve correctness; in this eval it mostly did not, and TDD specifically underperformed. If a skill is going to help, the author's own experience suggests it should actively nudge the agent away from its default failure patterns (verbose but low-value tests, superficial use of a named technique) rather than read like a tutorial explaining the technique, which is closer to how the underperforming skills were written.

How solid is it

The comparison is grounded in a fairly large run count: an average of 80 runs per condition and effort level (medium and xhigh) on the Zstd-in-Rust task, scored by the fraction that passed a hidden test suite, plus 40 runs per condition on a separate IMAP RFC task and a few individual runs on other RFCs, with results described as not materially different across tasks. The source text as captured does not give an absolute pass-rate figure for any individual condition, only relative comparisons such as "did well above average" or "a little better"; it is also truncated before the article's detailed per-condition write-ups and its conclusion.

Risks and caveats

The source does not identify the article's author by name, referring only to "we"/"I", and gives no explanation of what "GPT-5.6 Sol" specifically refers to beyond being the model used inside Codex. The author's closing question, about why AI labs have not built reinforcement-learning environments to train testing skill directly, is posed openly and not answered or resolved in the captured text; it should be read as speculation rather than a finding of the eval itself.

“AI agents' approach to testing, more or less: Take the pathological cases dreamed up by someone objecting to mocks 15 years ago, without ever having actually used mocks. Naive dreams of excessive mocking. Make those pathologies the backbone of your testing strategy.”

— Gary Bernhardt, quoted in the article