Glonce

Paper replicates METR developer study, says AI time savings must be counted with errors

researchThe Glonce editorsarXiv🔥62Glonce rating62Impact50Novelty60Relevance90Surprise45Players35Credibility75

An arXiv paper by a single author asks what it means for AI assistance to save time. It contrasts two views. On the stopwatch view, time saved is whatever the clock records. On the ledger view, which the author argues for, time saved is a counterfactual that has to be read next to the errors the work lets through.

The argument has four parts. First, the author defines a model-free pair of quantities, time saved and excess escaped defects, and one decision number built from them: the break-even defect cost. Second, a baseline model of checking makes three predictions. Until the model is good enough not to check, a calibrated worker's escaped-defect rate does not depend on how good the model is. Checking time grows with the logarithm of the model's error rate. And over-trust buys minutes on the clock at a larger cost in total minutes.

Third, the paper looks at what randomized studies can and cannot identify, and replicates METR's developer study from its public data. The same 246 issues support a slowdown from AI of 19% or 24%, depending on the estimand, and 34% without adjusting for a chance imbalance. AI helped on somewhere between 3% and 82% of those issues, which is a range of bounds rather than a point estimate. The paper also restates the published record with stated bounds in place of imported point assumptions. In GDPval's own review-every-output workflow, review time alone caps GPT-5's speedup at 3.6x. The consultants outside the frontier were faster, and more often wrong.

Fourth, the author tests two of the model's predictions in a small pre-specified experiment. AI reviewers check AI drafts of the same kind of brief, built from SEC filings made after the models' training data were collected. When the reviewer was told the drafter's true error rate, it checked more as that rate rose, but all-or-nothing, switching near the point where the costs in its prompt break even. When it was told that the drafter erred ten times less often than it did, it checked 7 fewer of 9 figures and let a wrong figure through in 22.5% of briefs instead of none. The author's summary: faster and worse, entirely through reduced effort.

Key facts

  • The paper argues for a ledger view of AI time savings: a counterfactual read next to the errors the work lets through, not just what the clock records.
  • Replicating METR's developer study from public data, the same 246 issues give a slowdown of 19% or 24% depending on the estimand (34% without adjusting for a chance imbalance), and AI helped on between 3% and 82% of them.
  • In GDPval's review-every-output workflow, review time alone caps GPT-5's speedup at 3.6x.
  • In a small pre-specified experiment, an AI reviewer told the drafter erred ten times less often than it did checked 7 fewer of 9 figures and let a wrong figure through in 22.5% of briefs, versus none in the comparison condition.

Why it matters

Claims about AI saving time usually rest on what a clock records. This paper says that is the wrong yardstick on its own. If speed comes with more mistakes that escape review, the minutes saved can be offset elsewhere, and the baseline model says over-trust buys minutes on the clock at a larger cost in total minutes. The paper proposes a pair of quantities, time saved and excess escaped defects, and a single decision number, the break-even defect cost, to put both on one page. Its replication of METR's developer study shows how much the answer can move with the estimand: 19%, 24% or 34% slowdown from the same 246 issues.

Who it affects

Anyone who measures or reports AI productivity gains: researchers running randomized studies of AI assistance, teams evaluating coding or document assistants, and readers of benchmark results such as GDPval. It also bears on people who build AI review or checking steps, since the experiment suggests that a reviewer's belief about the drafter's reliability changes how much it checks.

How to use it

The practical reading is a question to ask of any time-saving claim: what happened to errors that escaped? The paper's framework asks for both time saved and excess escaped defects, and combines them into the break-even defect cost. The source does not give that cost's value or a worked procedure beyond the definitions summarised here, so the full paper is the place to look for implementation detail. For GDPval-style workflows, the 3.6x cap shows that when every output must be reviewed, review time sets a ceiling on speedup no matter how fast the model drafts.

How solid is it

This is a single-author arXiv paper, and the source text does not name the author or institution. No submission date or publication venue is given; peer review status is not stated. The METR replication uses public data and reports ranges instead of point estimates, which fits the paper's own emphasis on bounds. The experiment is described as small and pre-specified, with briefs built from SEC filings made after the models' training data were collected, which addresses the worry that the models had seen the material. The source says the experiment tests two of the baseline model's predictions but does not say which two.

Risks and caveats

The experiment uses AI reviewers checking AI drafts, so it does not show how human workers behave. The number of briefs and which models acted as drafter and reviewer are not stated. The two estimands behind the 19% and 24% slowdown figures are not named, and the source does not give METR's original published slowdown figure. The 3.6x cap is described for GPT-5 only. The 3% to 82% range is wide by design: it states what the data can and cannot identify, not a best guess. The baseline model's claims, such as logarithmic growth of checking time, are properties of the author's model, not measured facts about all workplaces.

“faster and worse, entirely through reduced effort”

— From the paper's abstract, describing the AI reviewer told the drafter erred less often