A harness design for autonomous coding agents that can't grade itself
The author, writing on their own blog and submitted to Hacker News by user jarredkenny, describes a harness for running autonomous coding-agent development loops past the point where "give the agent a plan, let it work until the tests pass" is enough. That simple loop only protects failures the tests already describe; it misses a screen that looks complete but stores nothing, an agent that lands the right answer for the wrong reason, or a scorer that rewards behavior no user wants, and more agent turns make those cheaper to produce rather than less likely.
The design keeps two agents strictly separate. A development agent changes the feature and knows the code and the expected result. A product agent, standing in for the eventual user, gets a fresh conversation and only the tools the product itself exposes; if the two shared context, the product agent could succeed using knowledge a real user would never have, and the test would be worthless. Around them sit a driver that exercises the feature the way a user would (browser, API, command or hardware simulator), a scorer that reads what happened, and a controller that picks the next gap to close from that evidence.
Each round follows a fixed sequence: reset the fixture and confirm the system is healthy, run a deterministic "floor" of tests and effect checks, send one pre-approved request through a fresh interaction, record the answer, tool calls, rejected actions, visible result and persistent effects, classify the single largest gap, close it across every layer it touches (data model, tool contract, runtime, agent instructions, interface, effect check), and add a regression check before moving on. The floor only ever starts green and can only regress into a new job; it does not tell the loop what to build next, which is why the driven request and the scorer exist separately from it.
What the loop is allowed to do to itself is bounded by three levels of authority. "Free" files are ordinary implementation levers the loop can change and measure inside its assigned scope. "Propose" files cross a boundary: the loop can record evidence and suggest a change, but a person decides. "Frozen" files, the approved request corpus, fixtures, existing checks, score rules and judgment rubric, define the exam and are off limits. The loop can add a new regression check for a failure it observed, but it cannot weaken an existing check, lower a threshold, or change a product lever and the measure of that same lever in the same round, which would let it grade its own repair. A learned judge, such as a visual one, counts only as evidence until its rankings agree with repeated human rankings.
Progress lives in the repository rather than in any one conversation, since agent sessions end and context compresses. A small package under goals/ holds LOOP.md, which changes when the process changes, and STATE.md, which changes after every round, so a new agent session can read the goal, procedure, state and evidence and continue the same decision rather than starting over. The author recommends starting small: three approved requests, one fixture, one score command, one loop file, one state file, and a three-round budget, adding a new control only once a failure justifies it. At the end of each small batch, a person decides whether to continue the queue, redirect it, or stop; the development agent can propose a new direction but cannot decide on its own that its result is good enough. The author notes most tickets do not need any of this: if complete tests already describe the work, giving the agent the tests and letting it finish is enough.
Key facts
- The harness splits work across two separate agents: a development agent that knows the code and the expected result, and a product agent that gets only a fresh conversation and the tools the product itself exposes, so the two cannot share context that would make the test worthless.
- Each round runs a fixed sequence: reset the fixture, run a deterministic 'floor' of tests, send one pre-approved request through a fresh interaction, and classify and close the single largest capability gap across every layer it touches.
- An authority model with three levels limits what the loop may change on its own: 'free' files it can edit and measure freely, 'propose' files where it can only suggest a change for a person to approve, and 'frozen' files (the approved request corpus, fixtures, existing checks, score rules and rubric) it cannot touch.
- The loop can add a new regression check but cannot weaken an existing one, lower a threshold, or change a product lever and the measure of that same lever in the same round, specifically to stop it from grading its own repair.
- State persists outside any single conversation in two repository files, LOOP.md for the process and STATE.md for progress after each round; the author's recommended starting point is three requests, one fixture, one score command, one loop file, one state file and a three-round budget, with a person deciding after each batch whether to continue, redirect, or stop.
Why it matters
The essay targets a specific failure mode: once an autonomous coding agent is set loose on a capability that isn't already described by an existing test suite, the failures that matter move outside the tests entirely, a finished-looking screen that stores nothing, a right answer reached the wrong way, a scorer that rewards behavior nobody wants. Running the agent longer does not fix this; it makes those failures cheaper and faster to produce. The argument is that the harness surrounding the agent, not the agent's prompt, is what needs deliberate engineering once tests stop being sufficient on their own.
Who it affects
This is written for people building agentic coding products or internal agent-driven development pipelines who have already outgrown a simple 'loop until green' setup. It addresses the harness design layer (fixtures, drivers, scorers, controllers, authority rules) rather than end users of any tool, and names no vendor, model or specific codebase; it is a general architecture proposal, not a product announcement.
How to use it
The recommended starting setup is small: three approved user requests, one fixture, one score command, one loop file and one state file, run in three-round batches. Each round resets the fixture and checks system health, runs the deterministic test floor, sends one request through a fresh interaction, records the visible result and any persistent effects, classifies the single largest capability gap, closes it across whichever layers it spans (data model, tool contract, runtime, instructions, interface, effect check), and adds a regression check before the next round. Process state lives in two files under a goals/ folder in the repository, LOOP.md for the procedure and STATE.md for progress, so a new agent session can resume without the previous session's reasoning.
How solid is it
The piece is a first-person design essay posted to the author's own blog (jx0.ca) and submitted to Hacker News, presented as a follow-up to the author's earlier post 'The Convergence Problem.' It drew modest engagement, 26 points and 3 comments at last check. No benchmark, before/after measurement, deployment name, or code sample accompanies the claims; the argument is a coherent description of a proposed mechanism, not evidence that it outperforms a simpler agent loop in practice.
Risks and caveats
The design adds real overhead, two separate agent roles, a fixture, a scorer, a three-level authority model and two control files, for a problem the author says most tickets don't actually have: when complete tests already describe the work, the recommendation is simply to hand the agent the tests. The safeguards against the loop grading its own work (the free/propose/frozen split, the rule against touching a lever and its own measure in one round) depend entirely on people honoring the review boundary at each batch; the essay does not address what happens if that human step is skipped or rushed.
“The development agent can propose a new direction. It cannot decide that its own result is good enough.”
— the author