Across 37,623 real pull requests, agent-written code was not the maintenance burden many expected. Which vendor's agent wrote the code predicted revert rates and security smells better than the agent-versus-human split did.
Coding agents now open pull requests under their own name. The AIDev dataset, a public collection of such pull requests, records close to a million of them in under a year. Most evidence about the quality of this code comes from benchmarks, which show what a model can produce on an isolated task, not what happens after a maintainer merges the code. The author asks a narrower question: what does it cost a project to keep agent-written code for the 90 days after merge?
The corpus is 37,623 pull requests from 2,807 public GitHub repositories with more than 100 stars, opened between December 2024 and July 2025. Each pull request carries a provenance label, a record of who wrote it: OpenAI Codex, Devin, GitHub Copilot, Cursor, Claude Code, or human. The five agents wrote 33,596 of them. The human baseline is 4,027 pull requests from the same 810 repositories and the same time window, with a cap per repository so no single project dominates.
The author enriched every pull request through the GitHub API and cached all 58,792 responses. For merged pull requests, the pipeline fetched the commits that touched the most-changed file during the next 90 days and flagged revert commits by their message. This gives four post-merge signals: follow-up commits, churn per changed line (later edits to that file, relative to the size of the change), revert rate, and human review activity.
Code quality is measured on the added lines of the diff only, for Python, JavaScript, and TypeScript. That subset is 8,933 pull requests and 1.35 million added lines. A catalog of regular expressions matches eight weakness classes from the Common Weakness Enumeration, a public list of software weakness types. Examples are hardcoded credentials, SQL built by string concatenation, shell injection, and use of eval. Maintainability counts cover TODO markers, long lines, comment lines, branch keywords, and nesting depth. A regex hit is a smell, not a confirmed vulnerability.
The metrics are mostly zero with a few very large values, so the author uses statistical tests that do not assume a normal distribution. Each family of comparisons is adjusted so that many tests do not produce false positives by chance. Binary outcomes such as reverts are reported as odds ratios against the human baseline. An odds ratio below 1 means the agent's pull requests showed the outcome less often than human pull requests did.
Reverts split by vendor, not by agent versus human. Human pull requests were reverted within 90 days 11.5% of the time. Codex pull requests were reverted 6.1% of the time, an odds ratio of 0.50. Devin pull requests were reverted 14.5% of the time, an odds ratio of 1.31. Copilot, Cursor, and Claude Code were statistically indistinguishable from humans.
Pooled agent code carried fewer security smells. A pull request from any agent contained at least one smell 2.9% of the time, against 4.6% for humans, an odds ratio of 0.63. Two classes drive the difference: hardcoded credentials appeared in 0.9% of agent pull requests versus 2.2% of human ones, and eval or exec calls in 0.2% versus 0.8%. No weakness class was over-represented in pooled agent code after correction. Smell density per line, as opposed to presence, was similar across all groups. By the standard thresholds for effect size, all five per-agent differences were negligible.
One agent is an outlier on smells, and size is the likely reason. Claude Code pull requests contained a security smell 9.5% of the time, about twice the human rate, and carried hardcoded-credential smells in 3.3% of analyzed pull requests. Codex and Cursor were at 0.5% on that class, a fourfold spread between products. But the median Claude Code pull request changed 495 lines, against 52 to 96 lines for every other group. A larger diff has more chances to contain any pattern. A size-stratified check found the pooled agent-versus-human difference in smell density only in the largest size bucket. The author names size as the confound to watch and cannot remove it with the data available.
No agent imposed a higher per-line maintenance burden. On churn per changed line over 90 days, Claude Code had the lowest value of any group, with a medium-sized effect against humans. Copilot and Cursor were also below the human level. Codex and Devin matched humans. On raw follow-up commits, files touched by Copilot saw the least subsequent activity of any group. The large rise in churn that some industry commentary predicted did not appear in this window.
Review effort is uneven. Review records exist only for agent pull requests, so this comparison is agent against agent. Copilot pull requests received the most review: 3.6 human reviews and 0.43 change requests per pull request on average, plus the most bot reviews. Claude Code pull requests waited a median 12.6 hours for a first human review, against 1 to 4 hours for the others. The author attributes this to their much larger size.
Agents were not randomly assigned to tasks. Teams choose which agent to use and for what, so vendor differences mix code quality with task mix. Codex wrote 58% of all agent pull requests; Claude Code wrote only 459, so its intervals are wide. Static analysis covered 23.7% of pull requests, and review coverage ranged from 5.4% for Codex to 51.2% for Copilot. Churn is measured on the most-changed file only, and revert detection by commit message misses silent rewrites. The window is 90 days in popular public repositories, so results may not transfer to private codebases or later agent versions. One early path-traversal pattern matched relative-import strings and inflated that class about 150-fold before the author tightened it.
Treat the choice of agent as a measurable engineering decision, not a category. If you compare vendors, compare them on post-merge outcomes in your own repositories: revert rate, follow-up commits on the touched files, and time to first review. The released pipeline computes all of these, so you can re-run it on your organization's data.
Control pull request size at the harness level. The one agent that looked worst on smells and review latency also produced diffs about eight times larger than the rest. A cap on changed lines per pull request, or a rule that splits large changes, removes the confound that this study could not.
Keep a cheap smell scan in the merge gate. The regex catalog here runs on added diff lines only and costs almost nothing. Hardcoded credentials and eval-style constructs were the two classes where agent code differed from human code, in both directions across vendors. A gate that blocks those two patterns targets the classes where the groups differed most.
Plan for review capacity. Copilot pull requests already consume several human reviews each, and bot reviews outnumber human reviews for some agents. As agent volume grows, human attention becomes the scarce resource.