▲ AI productivity
Does AI make developers slower? What METR measured, and what it corrected in 2026.
By César Soto · 9 min read
▲ In this article
▲ In short
- In July 2025, a randomized controlled trial by METR with 16 experienced developers found that allowing AI tools increased the time to complete tasks in repositories they already knew by 19%.
- In February 2026, METR published data from a larger study (57 developers, more than 800 tasks) pointing to a speedup, with confidence intervals that include zero, and warned that its own estimate is likely a bad proxy for the real effect.
- Since there is still no clean measurement of AI's effect on developer productivity, the useful move for a company is to measure its team's delivery with three indicators: deliveries per week, idea-to-production time and requirements that QA sends back.
Short answer: nobody knows yet. METR's 2025 study measured 19% more time with AI across 16 developers, and in February 2026 its authors published new data that points the other way and that they do not consider reliable either. Whether your team delivers more is something you can measure in a few weeks with the history you already have.
The 19% keeps showing up in articles without the February update. I am CTO at Simón, where we run the platform with agents, and I help other teams put that way of working in place; this is what is worth knowing before you decide anything based on the study.
What did METR measure in 2025?
METR measured how long 16 experienced open source developers took to resolve 246 real tasks from their own repositories, with and without AI. Each task was randomly assigned to one of the two conditions, and with AI they mostly used Cursor Pro with Claude 3.5 and 3.7 Sonnet. With AI allowed, tasks took 19% longer.
The finding that got the most attention was perception. Before starting, the developers expected AI to cut their time by 24%, and afterwards they believed it had cut it by 20%. Economists and machine learning experts had forecast cuts of 39% and 38%.
The scope was narrow by design: people with about five years on mature projects they knew well, using early-2025 AI.
What did METR correct in February 2026?
On February 24, 2026, METR announced it was changing its experiment design because the new data was no longer a reliable signal. That second study started in August 2025 with 57 developers, 143 repositories and more than 800 tasks.
There are two estimates. For the 10 developers who returned from the original study, AI reduced time by 18%, with a confidence interval from 38% less to 9% more. For the 47 new ones, the reduction was 4%, with an interval from 15% less to 9% more.
The problem METR reports is selection. Between 30% and 50% of participants said they stopped submitting some tasks because they did not want to do them without AI. Hourly pay also dropped, from USD 150 to USD 50.
That is why METR wrote that its central estimate is "likely a bad proxy for the real productivity impact of AI tools on these developers". It added that the true effect could be larger, because selection leaves out precisely the tasks that depend most on AI.
| 2025 study | 2026 update | |
|---|---|---|
| Sample | 16 developers, 246 tasks | 57 developers, 143 repositories, more than 800 tasks |
| Effect on time | +19% (interval +2% to +39%) | −18% for returning developers (−38% to +9%); −4% for new ones (−15% to +9%) |
| METR's reading | AI slowed them down in that context | Unreliable signal; redesigning the study |
Both 2026 intervals cross zero
Change in task time when AI is allowed. Dot: central estimate. Bar: 95% confidence interval.
Why METR distrusts 2026
- People unwilling to work without AI stopped taking part.
- 30 to 50% held back tasks they did not want to do without AI.
- Pay dropped from USD 150 to 50 per hour.
What you can read from it
- With the 2026 data, the direction of the effect is unclear.
- METR calls its estimate “likely a bad proxy” and is redesigning the study.
- No number from a study replaces measuring your own team.
So does AI make developers faster or slower?
With what has been published up to November 2026, nobody can answer that with a clean number. Industry reports do not settle it either, because each one measures something different, with its own bias:
- LeadDev surveyed 617 engineering leaders in March 2025: only 6% reported a significant productivity increase. It measures those leaders' perception.
- DORA 2025 (Google Cloud, nearly 5,000 professionals) found that 90% use AI at work and that adoption is associated with higher delivery throughput and lower stability.
- Faros AI, with telemetry from 22,000 developers, reports that median time in review rose 441.5% in high-adoption teams. Faros sells a measurement platform, so this is vendor data.
DORA concludes that AI amplifies whatever the team already had, and I agree with that reading.
What each study measures before you quote it
None of them measures what a CEO pays for: how much reaches production, and at what quality.
Why doesn't a time-per-task study answer a CEO's question?
Because METR measures how long one person takes to close a task, and between that task and production sit the PR review, pending product decisions, QA and the release. That is where work gets stuck once code comes out faster.
Today Simón ships 2 to 3 releases per week on a B2B platform run with agents, without hiring more developers. To get there we worked mostly around the code: specs in the repo that the agent reads to make decisions, automated reviewers on every PR, human checkpoints where the risk calls for one, and bugs that come back into the flow as PRs.
A concrete case: QA kept sending back incomplete requirements that did not meet all their acceptance criteria. I added an agent that reads those criteria on every PR and looks for evidence of each one, and send-backs dropped to practically zero. A time-per-task study would not have registered that change, because the saving shows up after the PR, in the rework QA no longer has to return.
I explain it in more detail in what a harness is and in the PeakSyn method.
What should you measure in your team instead of waiting for the next study?
Measure the team's delivery with three indicators that come from the history you already have: deliveries per week, idea-to-production time and requirements QA sends back. They are the ones I use with the teams I work with, and none of them requires buying a tool.
| Indicator | What it counts | Where it comes from | Common mistake |
|---|---|---|---|
| Deliveries per week | Changes that reach production and that a user can use | Release notes, version tags or the "in production" column on the board | Counting PRs or deploys, which rise with AI even when users get nothing new |
| Idea-to-production time | Median days from when a requirement is ready to build until it reaches production | Tracker dates (Linear, Jira) and deploy dates | Starting the clock when someone opens the editor, which leaves out time waiting for decisions |
| Requirements QA sends back | Share of requirements QA returns for not meeting their acceptance criteria | Status changes in the tracker | Resolving the send-back in a chat, where it is never recorded |
I left out lines of code, PRs opened and the share of code written by AI because they measure usage. In 2025 Faros reported almost twice as many merged PRs in high-adoption teams, and that number alone does not tell you whether any customer received something new.
Three numbers that come from your own tools
You can pull them from the tracker and deploy history in an afternoon. Read them as a 4 to 6 week trend.
Deliveries per week
WHAT IT COUNTSChanges that reached users, under a written definition
WHERE IT COMES FROMDeploy or release history
HOW TO READ ITThe trend, never a single week
Idea → production
WHAT IT COUNTSTime from when the requirement is approved until it is in production
WHERE IT COMES FROMStatus change dates in the tracker, plus deploys
HOW TO READ ITIf it rises, look for where work is waiting
QA send-backs
WHAT IT COUNTSRequirements QA returns because they miss their criteria
WHERE IT COMES FROMTracker: tasks that go back to development
HOW TO READ ITIt should drop once a reviewer checks the criteria
- 1Baseline from the last 8 weeks
- 2What counts as a delivery, in writing
- 3Every process change, with its date
- 4Weekly review with the CEO and tech lead
- 5Always team level, never per person
How do I start measuring this week?
Pull the baseline from what is already in your tools and review the three numbers every week for six weeks.
- Pull the baseline for the last 8 weeks. The tracker and deploy history already have the dates.
- Write down what counts as a delivery. One line in the repo keeps the number from changing depending on who pulls it.
- Log every process change with its date. If you add an automated reviewer or change who approves a PR, you will be able to read the curve later.
- Review the numbers every week with the CEO and the tech lead. Ten minutes is enough; what matters is the four-to-six-week trend.
- Keep the indicators at team level. Once they are used to grade a person, the number starts getting dressed up.
What do you say when someone quotes the 19%?
Acknowledge that the study is serious and explain what happened in February 2026, when its authors published data pointing in another direction that they do not consider reliable either. Then bring the conversation to your team's three numbers.
What I keep is the perception finding. If 16 experienced developers believed they were going faster while taking longer, any team can be in the same spot without knowing it. I distrust the 19% when it is used to slow adoption down, and the "10x" promised without a baseline, because neither comes from measuring an actual team.
If you want to work out how to pull your team's baseline and what would move those three numbers, book a 30-minute conversation. You leave with a clear next step, with me or without me.
Frequently asked questions
Does the METR study prove that AI makes all developers slower?
No. It measured 16 experienced open source developers, in repositories they had known for years, with early-2025 tools. In that context, they took 19% longer with AI. Applying that result to a product team with other tools and other code goes beyond what the study measures.
Did METR retract the 2025 study?
It did not use that word. On February 24, 2026 it published data from a larger study, explained why it no longer considers that data a reliable signal (developers withholding tasks they did not want to do without AI) and announced that it is changing the experiment design. The 2025 paper is still published on arXiv.
What does it mean that the confidence interval includes zero?
That the data is compatible with AI speeding up, slowing down or not changing a task's time. In the 2026 update, the intervals run from 38% less to 9% more time and from 15% less to 9% more, so the study cannot state the direction of the effect.
How long does it take to see whether AI improved a team's delivery?
With a baseline from the last 8 weeks, four to six weeks of weekly measurement usually give a readable trend in the three indicators. Less time than that gets confused with the normal variation of a work cycle.
Why not measure the share of code written by AI?
Because it measures how much the tool is used. A team can generate most of its code with AI and deliver the same if review, decisions or QA are still the bottleneck. Delivery indicators show whether something reached the customer.
References
- Joel Becker, Nate Rush, Elizabeth Barnes and David Rein, “Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity”, arXiv:2507.09089, July 2025
- Joel Becker, Nate Rush, Tom Cunningham, David Rein and Khalid Mahamud, “We are Changing our Developer Productivity Experiment Design”, METR, February 24, 2026
- LeadDev, “AI coding assistants aren't really making devs feel more productive”, June 11, 2025
- Google Cloud, “Announcing the 2025 DORA Report”
- Faros AI, “AI Acceleration Whiplash”, April 12, 2026 (vendor telemetry)
- Faros AI, “The AI Productivity Paradox Report 2025”, July 2025 (vendor telemetry)
- César Soto, “What a harness is and how it makes an AI agent trustworthy”, PeakSyn, October 2026
- PeakSyn, “The method v2026.09”
Want to apply it in your team? Let's talk for 30 minutes.
Free, on video, with César Soto.
Book a conversation