Harvard working paper: AI coding agents moved the bottleneck to code review

Harvard working paper: AI coding agents moved the bottleneck to code review

ComplexDiscovery reports on a Harvard working paper by Ph.D. candidates Fiona Chen and James Stratton. It measures what happens at firms that adopted AI coding agents, across 725,938 workers at 718 companies. Engineers added about 30 percent more code. Finished work did not measurably rise, and getting code through review took 49 percent longer.

The data come from Jellyfish, a company that sells an analytics platform tracking engineering activity, joined to LinkedIn employment records collected by Revelio Labs. The sample runs from January 2021 to March 2026 and covers Jellyfish client firms that consented to research use of their data. The tables list 68.3 million Jira issues, 200.9 million commits, 49.1 million pull requests and 21.3 million calendar events, though only 213 of the 718 firms supplied calendar data. The design is a staggered difference-in-differences comparison of adopting firms against firms that had not yet adopted. The authors say adoption timing was shaped mainly by when firms learned about the tools and by how long procurement took (sourcing, legal and security approvals, pilots). One interviewee's firm approved a tool in about a day; another interviewee put average approval time at six to 12 months. The authors acknowledge that larger firms adopted earlier and control for firm size. If some hidden difference remains, they say, it would more likely make AI look more productive than it is, not less.

On the coding side, lines of code added rose by 1,494 per worker-month against a baseline of 5,017, about 30 percent. Commits rose 20 percent and pull requests rose 23 percent. The authors note these estimates capture the added effect of agents at firms that already had AI coding assistants, not the effect of AI from a standing start.

Output is where the gain stops. The paper counts finished work as Jira issues resolved, where an issue is complete once its code is deployed to production. Agents moved that measure by 0.12 issues per worker-month on a 3.67 baseline, with a standard error of 0.17. That is statistically indistinguishable from zero, and the 95 percent confidence interval rules out an increase above about 12 percent of the baseline. The researchers also tested whether engineers had shifted to larger tasks. They labeled a 0.5 percent sample of issues by size with GPT-4o and extended the labels to every issue with a text-embedding model. They found no shift in average issue size, and pull requests did not grow larger either.

The paper locates the shortfall in review. Time from pull request submission to merge rose 3.45 days on a 7.03-day baseline, a 49 percent increase. The share of pull requests receiving a formal change request rose by about 12 percentage points from a baseline of 13 percent, nearly doubling. Review comments per pull request rose 35 percent. The share of workers doing any code review rose 14 percent from a base of about 29 percent, while the share writing code showed no measurable change.

AI review tools did not remove the bottleneck. By March 2026 nearly 80 percent of the firms had used one, yet AI generated only 23.3 percent of review comments, and only 10.8 percent of pull requests received even one AI comment. The authors write that the bottleneck persists after firms adopt such tools. A junior software engineer interviewed in the paper described a project where only one senior engineer could approve changes, and could review the interviewee's code only once a week. After outages caused by AI-generated commits, new rules required senior approval for AI-generated code, adding more review bottlenecks.

The authors' model has two channels. In the first, faster drafting sends more code to a fixed pool of reviewers, a problem that moving labor into review can largely relieve. In the second, the rise in change requests and comments, which faster drafting alone does not explain, reads through the model as a sign of lower-quality drafts. Fixing that, the authors say, would require better technology.

Employment did not visibly move either. The estimates rule out a decline in overall employment larger than 2.9 percent, and a noisier estimate rules out a decline in engineering employment larger than 13.8 percent. The authors caution that these are short-run effects measured in partial equilibrium, which could mask job losses at competing firms, and that the engineering employment estimate shows upward pre-trends that make it hard to read causally.

ComplexDiscovery then ties the study to legal AI. Harvey's motion-to-dismiss workflow, co-developed with LexisNexis and announced Oct. 7, says a task that commonly takes 40 to 50+ hours is reduced to research and a first draft in a matter of hours, with human checkpoints for a lawyer. Cooley's Oct. 8 announcement describes a court-filing redaction agent, still in development with Google Cloud, where attorneys make all redaction determinations before filing. Neither announcement reports how long the review stage takes, and Cooley's carries no error rate. The article stresses that Chen and Stratton studied no legal work, so the software findings are a pattern in software, not a finding about law firms.

A separate NBER working paper, revised in September 2026, reports a similar falloff using data on over 500,000 GitHub developers. Its authors, Mert Demirer, Leon Musolff and Liyuan Yang, found a 240 percent cumulative rise in commits from autonomous agents shrinking to 30 percent for actual releases. Demirer and Musolff each previously held a postdoctoral position at Microsoft, which owns GitHub, and each now works as a paid research consultant for it.

Key facts

  • At firms that adopted AI coding agents, lines of code added rose by 1,494 per worker-month on a baseline of 5,017 (about 30 percent), but Jira issues resolved moved by only 0.12 on a 3.67 baseline, with a standard error of 0.17.
  • Pull request time to merge rose 3.45 days on a 7.03-day baseline (49 percent); the share of pull requests with a formal change request rose about 12 percentage points from 13 percent.
  • Nearly 80 percent of firms had used an AI review tool by March 2026, yet AI wrote only 23.3 percent of review comments and 10.8 percent of pull requests got an AI comment.
  • The study covers 725,938 workers at 718 Jellyfish client firms from January 2021 to March 2026; it is a job-market paper by Fiona Chen and James Stratton, with no indication of completed peer review.
  • The estimates rule out an overall employment decline larger than 2.9 percent, but they are short-run and the engineering employment estimate shows upward pre-trends.

Why it matters

The study offers measured evidence that writing code faster is not the same as shipping more. Code written, commits and pull requests all rose, yet resolved issues did not measurably change, and the extra volume landed on reviewers. Merge times lengthened by 49 percent, change requests nearly doubled as a share of pull requests, and AI review tools covered only a minority of comments. A separate NBER paper on over 500,000 GitHub developers reports a similar falloff: a 240 percent cumulative rise in commits from autonomous agents shrinking to 30 percent for actual releases.

Who it affects

Engineering teams and the managers who buy coding agents, since review capacity appears to set the pace once drafting speeds up. Senior engineers who approve changes feel it most: the paper's share of workers doing any code review rose, and an interviewee described one senior approver reviewing code once a week. ComplexDiscovery also points to eDiscovery, privacy and compliance professionals and legal AI buyers, because tools such as Harvey's motion-to-dismiss workflow and Cooley's redaction agent keep a lawyer at a comparable review step.

How to use it

ComplexDiscovery's suggestion for legal AI buyers is to measure approved work product, revision rates and reviewer time alongside drafting output, rather than drafting output alone. It also notes that the paper identified agent use partly from bot accounts tied to AI tools and Co-authored-by lines in commit messages and pull request descriptions, so AI authorship can leave a metadata trail that retention schedules and legal holds may need to reach. The article advises watching for a revised version of the paper and for any legal vendor that publishes review-stage figures. The authors' two-channel model also suggests asking which problem dominates: too few reviewers, which moving labor into review can largely relieve, or lower-quality drafts, which they say would need better technology.

How solid is it

The paper is a job-market paper circulated by Chen, and nothing in the version ComplexDiscovery reviewed indicates journal acceptance or completed peer review. The current version is dated Aug. 4, 2026; the first was dated Jan. 7. The scale is large (718 firms, 725,938 workers), and the authors tested one obvious alternative explanation, larger tasks, finding no shift in issue size. They say any remaining hidden difference between early and late adopters would more likely flatter AI than hurt it. The data come from a single analytics vendor's consenting clients, and only 213 of the 718 firms supplied calendar data. This account comes through a secondary site, not the paper itself.

Risks and caveats

The estimates capture the added effect of agents at firms that already had AI coding assistants, not AI from a standing start. The employment results are short-run effects in partial equilibrium, which could mask job losses at competing firms, and the engineering employment estimate shows upward pre-trends. The 14 percent rise in the share of workers doing code review is stated by the source without saying whether it is relative or in points. The study covers software, not legal work: document review is not code review, and the article says the two studies point to a pattern in software, not a finding about law firms. The NBER paper's authors Demirer and Musolff disclose paid consulting ties to Microsoft, which owns GitHub.

“After some outages that were caused by AI-generated commits, there were also new rules requiring senior engineer approval for AI-generated code, which created even more review bottlenecks”

— A junior software engineer interviewed in the paper (lightly edited for clarity by the authors)