News · 2026-11-18
Coding agents sped up pull requests as review delays grew in a large firm study
Coding-agent adoption was followed by more submitted changes and longer elapsed code-review times in a study of 655 firms. The researchers estimate a 56.1% increase in time from pull-request submission to merge, suggesting that review capacity can become a bottleneck when code production accelerates.
Key facts
- Fiona Chen and James Stratton’s working paper is dated November 14.
- The sample includes 655 firms and 445 million recorded events.
- Pull requests opened per worker rose an estimated 35.7% after 12 months of agent adoption.
- The 56.1% review measure is elapsed calendar time, not hours of reviewer labor.
A pull request is a proposed change that colleagues can inspect before it is merged into a shared codebase. If agents help engineers produce changes faster, the review queue can grow even when reviewers work just as hard. Picture a kitchen that can prepare twice as many dishes while the same small team must inspect every plate before it leaves.
Chen and Stratton’s paper, titled “Artificial Intelligence in the Firm: Bottlenecks in Software Production,” combines records from software repositories, Jira and calendars at firms that opted into research use of Jellyfish analytics data. It compares firms adopting at different times, adjusting for firm and month effects and other factors. This design can illuminate changes around adoption, but the firms were not randomly assigned agents. Its causal interpretation relies on assumptions about what would have happened to later adopters in the absence of adoption.
The authors estimate pull requests merged per worker rose 26.0%, while submission-to-merge time increased by 1.99 days. They also find increases in waiting for a first human comment and in time after the first comment. These components use a subset of pull requests with comments and should not be added to reproduce the all-request total. More comments and longer time after review begins are consistent with additional review effort, but neither directly measures reviewer hours or defects.
The distinction matters because several mechanisms can produce the pattern. An influx of changes may overwhelm reviewers; generated drafts may require more scrutiny; or teams may change approval rules at the same time. The study does not cleanly divide the delay among these possibilities. It also finds no significant change in its bug-report measure, which does not rule out quality changes that the measure misses.
An important counterexample is already inside the study: assistant adoption increased opened and merged requests without a statistically detectable increase in review duration. Separately, Jellyfish’s company release reports faster delivery outcomes at some high-adoption teams. Those promotional observations lack a comparable published queue measure and do not disprove the study’s average estimate. The paper’s Jira output estimates are imprecise: the issue estimate is borderline significant and the epic estimate marginally insignificant.
The community discussion included both predictions of an overloaded review queue and claims that specifications or code quality are the real bottleneck. Those opinions are not separate causal evidence. A useful team-level test would track incoming changes, waiting time, review time, defects and deployed outcomes together, rather than equating more generated code with more delivered value.
The authors also investigate why allocating more people to review may be difficult. They report that reviewers had previously contributed to the same codebase in 82.8% of reviews, compared with 17.2% under their calculated random-assignment comparison. Familiarity with the code and product can make a review more effective, but it also means an available engineer is not automatically an interchangeable reviewer for every change.
There is a further distinction between activity and outcomes. A pull request is a unit of workflow, not necessarily a unit of customer value. A larger number of small changes might be easier to review, or it might create more coordination overhead. Jira issues and epics offer a downstream measure, but the estimates here are uncertain and depend on how participating firms record their work. The paper’s data partner reviewed the manuscript for proprietary information, according to the authors, and the underlying firm-level data are not openly reproducible.
The study’s staggered adoption design is explained in our lesson on difference-in-differences. It relies on a comparison between earlier and later adopters, including an assumption about how they would have changed without adoption. That is more informative than simply observing that a queue grew after a tool launch, but it cannot remove every concurrent change. Independent replications with published measures of review labor and delivered quality would sharpen the result.
Key questions
How much longer did review take?
Does the paper prove AI-generated code is worse?
Cite this
APA
Ground Truth. (2026, November 18). Coding agents sped up pull requests as review delays grew in a large firm study. Ground Truth. https://groundtruth.day/news/coding-agents-review-queue-harvard-study.html
BibTeX
@misc{groundtruth:coding-agents-review-queue-harvard-study,
title = {Coding agents sped up pull requests as review delays grew in a large firm study},
author = {{Ground Truth}},
year = {2026},
month = {nov},
url = {https://groundtruth.day/news/coding-agents-review-queue-harvard-study.html}
}
Comments are replies to this story on Bluesky — reply with any Bluesky account to join in.