You are the worst possible judge of whether AI makes you faster

Ask a developer whether AI makes them faster and you get a confident answer. The confidence is not attached to anything.

Stanford's Software Engineering Productivity group asked 43 engineers to place their own productivity against the global average, then compared each answer to that engineer's measured output. The correlation was 0.17. People misjudged themselves by around 30 percentile points, and only one in three landed within a quartile of the truth.

That establishes something narrower than it first looks. It says engineers cannot rank their own output against their peers. It does not yet say whether they can feel a tool changing their own speed, which is a different question, and the one this post is about. For that you need to measure the same person twice.

METR ran a randomized controlled trial with 16 experienced open-source developers across 246 real tasks, in repositories they had worked in for an average of five years. These were not students. They were asked to predict the effect, then to report it afterwards, while a stopwatch ran.

A 39 point error, in the same people, about their own workday, on tasks they had just finished.

The usual reply to this is that you would notice. You have a sense of your own throughput, or you track your pull requests, or your team watches cycle time. Two problems with that. The Stanford engineers also had a career's worth of intuition and were off by 30 percentile points. And pull request counts are the one thing that reliably does go up, which is exactly why they are the worst number to check against. We come back to that below, with a company that raised its pull requests 13.6% and its actual output by 1%.

The study everyone cites, and the part they leave out

That result is real and it is also not the last word. In February 2026 METR published an update saying they had abandoned the experimental design. Developers refused to take part if it meant working without AI, even at $50 an hour. Between 30% and 50% of participants admitted they were steering the tasks they expected AI to be good at away from the study. One wrote that they avoided issues where "AI can finish things in just 2 hours, but I have to spend 20 hours." Agentic tools also broke the time-tracking, because developers started doing something else while waiting.

A weaker follow-up with the same developers pointed the other way, at roughly 18% faster, with a confidence interval running from 38% faster to 9% slower. METR calls this very weak evidence and says so plainly. It also draws a conclusion the people who quote the 19% tend to skip: the withdrawal mechanics make the original estimate likely a lower bound, and METR believes developers are probably more sped up now, in early 2026, than its early-2025 measurement showed. The newly recruited group came in 4% faster, with an interval running from 15% faster to 9% slower.

So the honest summary is that the best experiment in the field measured a slowdown, then withdrew its own method, and the reason it withdrew is itself a finding: developers will not give the tool up, and they route the easy work to it.

That matters for everything that follows, because the public argument about AI coding runs almost entirely on the one instrument we have just watched fail. Three positions are shouting at each other. The bubble that pops within a year, the 2030 date by which no human writes production code, and the middle that calls the whole thing assistance. All three are trading impressions.

So we went to the measurements instead. They support none of the three, because all three are arguing about the same step. They argue about whether the machine can write the code. That question is close to settled and it stopped being the expensive part.

The expensive part is now finding out whether the code is right.

Coldtea sells verification tooling. If the conclusion of this post is correct, we benefit.

The stage that absorbed the difference

A state-of-the-art review published in August 2026 assembled seventeen months of evidence on AI-assisted development across software engineering, human-computer interaction, labour economics, security, and education. It found the same dispersion we did and named six reasons for it. The same corpus also carries peer-reviewed field experiments reporting 26% more tasks completed per week.

Writing got faster. Checking got much slower. Whether a given study reports a speedup or a slowdown depends almost entirely on whether its measurement window is wide enough to include the checking.

The telemetry that fills that window comes from Faros AI, covering 22,000 developers across more than 4,000 teams, comparing each organisation's lowest and highest AI-adoption periods. Faros publishes two numbers here and they measure different things. Median time in review is up 441.5%, which is how long a pull request sits waiting. That is a queue, and queues lengthen when reviewers are outnumbered. The number this argument needs is the other one: average time spent in code review, up 199.6%. That is reviewer effort, and it is roughly half the size of the figure everyone quotes.

Faros is a vendor, which puts it low in our ranking, and part of any latency rise is volume rather than difficulty. Effort nearly doubling is the claim we will defend.

Nothing was removed from that pipeline. One stage got cheap and the work piled up against the stage immediately downstream of it. A team that responds by buying more seats is feeding the stage that already finishes in seconds.

The same review closes with a falsifiable conjecture: the gains are real on new code and shrink or reverse on mature codebases.

Stanford has already measured something close to it. Their group has been running a time-series study across the git history of more than 600 companies and 100,000 engineers. The gains sort cleanly by how new the project is and how hard the task is.

The top left cell is the demo. The bottom right cell is most people's job. That one square explains why the same technology produces triumphant blog posts and furious Reddit threads in the same week, and why the METR developers, working in repositories they had known for five years, were the ones who came out slower.

Stanford publishes a second grid crossing task complexity with how popular the language is. Both are described as orientative guidelines rather than promises, so we read them together and take the lower of the two bounds rather than averaging them.

One company, four metrics, no gain

The clearest single result we found is a case study from the same Stanford group. One company adopted AI coding assistants. Stanford measured four things before and after.

Rework is missing from that chart on purpose, because it is a multiple rather than a percentage and does not belong on the same axis. It rose 2.6 times, at p less than 0.01.

The visible activity went up 13.6%, which is the number a dashboard would show you. The quality went down. The value delivered did not move. And the rework, the number nobody was watching, went up 2.6 times.

If you measured this company by pull requests, the pilot succeeded. If you measured it by what shipped and stayed shipped, nothing happened, and the team spent considerably more effort producing that nothing.

What is actually happening to the code

GitClear analyzed 623 million code changes. Its maintainability signals move together and they all move the same way.

In 2022 developers were about twice as likely to refactor as to copy and paste. They are now about five times more likely to copy and paste. Refactoring, the act of moving code so the whole makes more sense, has nearly stopped.

Cross-file connectivity falling 35% means new code calls into the rest of the system less than it used to. Code is arriving in isolated blocks that do not connect to what is already there. And the long-term update rate falling 73% means old code is being touched less. Both are what you would expect if the people shipping the code do not know the codebase well enough to reach into it.

This is correlational and GitClear sells tooling in the space, so treat it as one witness rather than proof. It agrees with the others.

Security did not improve when the models did

Veracode has run the same 80 coding tasks against more than 150 models, in four languages, for two years. It is the longest-running measurement of this kind we found.

Forget the 45%. The number has not moved. Veracode reports security pass rates "stubbornly stuck at approximately 55%, virtually identical to where they stood two years ago." Two years of model progress, including the jump to reasoning models, bought nothing on this axis. Reasoning models reach 70% to 72%, which is better and still nowhere near shippable.

Capability went up sharply and this failure rate barely followed. The biggest jump in the window, reasoning models, bought 15 points and still leaves three failures in every ten.

OpenAI stopped reporting the benchmark everyone quotes

The case for full automation usually rests on benchmark curves. On 23 February 2026 OpenAI stopped reporting the most quoted one.

Its Frontier Evals team audited SWE-bench Verified and found that GPT-5.2, Claude Opus 4.5 and Gemini 3 Flash could each reproduce the original gold patch or the problem statement verbatim, from the task ID alone, with minimal prompting. A score you can recover from an identifier rather than from the problem is measuring memory.

Four and a half months later the same team audited the replacement. Of the 731 tasks in SWE-bench Pro's public split, their pipeline flagged 200 as broken and human annotation flagged 249. Between a quarter and a third of the tasks everyone is now scored on are defective.

That leaves the two headline numbers hard to read against each other. Claude Opus 4.5 scores 80.9% on Verified and 45.9% on Pro, which looks like a thirty-five point collapse. It is not, or not cleanly. The Verified figure comes from Anthropic's own scaffold and the Pro figure from Scale's standardised harness, and the same model on Pro scores anywhere from 50.2% to 55.4% depending on which agent system runs it. Correct for the defect rate on top of that and 45.9% of all Pro tasks is nearer two thirds of the solvable ones. Take the generous end of both corrections and the gap does not narrow, it disappears. Take the conservative end and it is around eighteen points.

We have also only corrected one side. Verified has its own defect problem, since OpenAI found flawed tests in 59.4% of the tasks it audited there, and correcting for that would widen the gap again rather than narrow it. We have left it uncorrected because those 138 tasks were chosen as ones a model kept failing rather than sampled at random, so the rate cannot be generalised.

Which end is right, we cannot say, and that is the finding. The most cited measurement of AI coding ability is contaminated at one end, defective at the other, sensitive to scaffolding throughout, and nobody can currently tell you what the distance between two of its scores means.

Both benchmarks also score discrete issue resolution. Most engineering effort goes into evolving systems nobody fully understands, and no current benchmark measures that.

The optimists and the exhausted agree on where the work went

What convinced me was not any single study. It was that the most bullish person in the field and the most worn down person in the field describe the same bottleneck in almost the same words.

That last one is the same finding arrived at by survey rather than by argument. Researchers questioned professional engineers twice, six months apart. 82% reported spending less time writing code. The work did not disappear. It moved into directing, evaluating and correcting output, which the authors named supervisory engineering work because there was no existing term for it.

The same study found something less comfortable. Perceived productivity held steady, with 84% reporting improvement at both points. But among the matched participants, the share reporting a worse developer experience in at least one dimension nearly doubled, from 14% to 27%. Flow state and cognitive load got worse while feedback loops got better.

People are getting more done and enjoying it less, which is roughly what you would expect if you replaced someone's creative work with someone else's code review.

Adoption is rising and trust is falling

Stack Overflow asked 2025's respondents both questions. The answers point in opposite directions.

Almost right is the top frustration, not wrong. Wrong code announces itself. Almost-right code compiles, passes review at a glance, looks like the surrounding style, and fails somewhere you were not looking. It is the most expensive possible failure mode, because the cost of catching it lands entirely on the reader.

The strongest case against this post

If verification is the bottleneck, the market should be solving it. The market is certainly trying. "There is an AI code review bubble" was one of the more popular things Hacker News read this year, and the comments were not kind.

The critics have the data on their side so far.

More than half of the machine's review comments get thrown away. That is not a junior reviewer learning a codebase, it is the leading product in the category, and 56.3% of what it says is noise.

The deeper objection is older than any of this. An engineer on Hacker News put Deming's version of it well: you cannot get good results from a bad process by adding quality control at the end. The US car industry tried exactly that. High defect rate, catch it in QA, rework. It lost to people who built the quality in.

If AI generation has a high defect rate and our answer is to inspect harder, we are repeating a mistake with a known ending.

We think both objections are correct about today and wrong about the conclusion, for one reason. The rejections were false positives, redundancy, out of scope, and misalignment with intent. Those are the failures of a reviewer that can see the diff and nothing else. Meanwhile the failure people actually report is code that is internally consistent, plausible, locally correct, and wrong about the system it just joined. A diff-shaped reviewer cannot see that, because the evidence is not in the diff.

So the CodeRabbit number does not measure whether machine verification works, only whether the patch is the right place to stand. The question that matters is not whether the code looks right. It is whether the software still does what it is supposed to do, which you find out by running it.

That is a claim we should be held to. Take a public set of repositories, have a diff-based reviewer and a behavioral one each produce findings on the same pull requests, and compare the rate at which maintainers accept each. If the behavioral findings land in the same acceptance range as the 36.4% above, the inspection objection wins and we are wrong. Nobody has published that measurement. We are running it on our own data, and our number should not be the one that settles it, because we sell the thing it would vindicate.

The gap between teams is widening faster than the average is moving

Arguing about the average productivity effect may be the wrong argument. Stanford tracked the spread between top-quartile and bottom-quartile teams over two years.

This is DORA's amplifier finding showing up in independent data. AI magnifies whatever system it lands in. A team with clear ownership and fast feedback gets faster. A team without them generates more unverified code and drowns.

What we found most useful is Stanford's summary of what the top quartile does differently. They run AI across the whole development lifecycle rather than only in the editor, and the examples given are continuous integration, testing and quality assurance. They run several agents in parallel instead of watching one think. And they measure their AI deployments as experiments rather than adopting on faith.

The first item on that list is a researcher with no product to sell arriving at the same conclusion as this post, which is the version of it we would trust more than ours.

Is it a bubble

Four layers, and the answer is different in each.

Hyperscaler capital expenditure is around $805 billion for 2026 on Morgan Stanley's estimate, against roughly $261 billion actually spent in 2024, with circular financing arrangements where the same dollars move between chip vendor, model lab and cloud provider several times. That layer looks like every capex bubble in history.

The model layer is real value with brutal commoditization and no durable moat at the API. Prices fall and capabilities converge.

The autocomplete-era tool layer is in trouble for an unglamorous reason. The feature is now standard inside every editor, so a company whose product is generation is selling something its suppliers give away.

The verification layer is underbuilt relative to the volume of code arriving, and currently full of products that do not work well, which is what an early market looks like from the inside.

A capital correction and a permanent technology are not alternatives. Railways were both.

What we expect, with dates

Five predictions, each with the measurement that settles it.

The greenfield conjecture holds when tested directly, meaning a study that separates new code from mature codebases finds gains on the first and neutral or negative results on the second. Settled by any properly controlled replication before end of 2027. We are 75% confident.

Security pass rates for generated code stay under 80% through 2027 without explicit security scaffolding, because two years of capability gains have already failed to move them. 70% confident.

Reviewing and specifying overtakes authoring as the largest share of engineering time in most organizations by end of 2027, measured by the time-allocation questions in the DORA, JetBrains and Stack Overflow surveys. 65% confident.

At least one AI coding company takes a valuation markdown of more than half while category revenue keeps growing. 60% confident, no date, because timing bubbles is not a skill we claim.

Comprehension loss shows up in an incident statistic before it shows up in a productivity statistic, meaning a major public postmortem names "nobody understood the generated code" as a contributing cause by end of 2027. 55% confident, and this is the one we would most like to be wrong about.

What we would do on Monday

Read faster than you write. The scarce skill is no longer producing a candidate solution, it is rejecting one quickly.

Measure the bottleneck rather than the output. Pull request count and lines shipped now measure the wrong end of the pipeline. Review latency, rework rate and change failure rate measure the end that binds.

Budget verification capacity in proportion to generation capacity. Every seat you add on the writing side adds load to the reading side, and only one of those two scales by buying more licenses.

Do not let the machine write the code nobody will ever understand. The comprehension research is the part of this literature I find hardest to shrug off. Students who built with an agent completed the task and then could not extend their own code without it. Low-effort interactions, copy-pasted prompts and auto-accepted edits predicted the worst comprehension. They preferred the agent anyway.

That last finding sits next to Stanford's payroll analysis, updated this month, which finds no widespread displacement but an employment gap for 22 to 25 year olds in the most AI-exposed occupations that has widened to 19%, software engineering among them. We are automating the task that used to manufacture senior engineers, and we have not replaced the mechanism that made them.

Method

We ranked evidence in tiers and worked down from the top: randomized trials and large-scale telemetry first, then peer-reviewed studies, then large surveys, then vendor research, then everything else. Every figure here traces to a primary source, listed at the end. The Stanford figures are circulating in summaries that get the quadrants wrong, so ours come from Denisov-Blanch's own slides.

The Stanford material comes from a conference presentation rather than a peer-reviewed paper, which places it below the trials in our ranking. We use it heavily anyway because it is the largest dataset anyone has published on this question and because its case study is the only place we found all four metrics measured on the same population.

The literature came from arXiv, filtered to work published between January 2026 and August 2026 on AI-assisted development, code review, technical debt and developer trust. The practitioner material came from Hacker News threads above 150 points over the last twelve months and from r/ExperiencedDevs top posts of the year. We could not access X directly, so we have used nothing from it.

The samples have known biases and we did not correct for them. Hacker News rewards contrarians. r/ExperiencedDevs skews skeptical, which one of its own popular threads captures in the title "AI is working great for my team, and y'all are making me feel crazy." Vendor research favours the vendor, including the vendor writing this post.

Safer Builders, Risky Maintainers is often cited as counter-evidence: agentic pull requests introduce fewer breaking changes than human ones, 3.45% against 7.40%. That figure covers code generation only. In the same paper agents break things at 6.72% on refactoring and 9.35% on chores, at or above the human rate the moment they touch existing code. Read whole, it is not counter-evidence, it is the greenfield-and-brownfield split arriving from a direction we did not expect.

Sources

Productivity measurement

• METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, and its preprint
• METR, We are Changing our Developer Productivity Experiment Design, the withdrawal and the lower-bound conclusion
• Yegor Denisov-Blanch, Will AI Replace Software Engineers?, the Stanford slides behind the quadrants, the case study, the self-assessment correlation and the quartile gap
• Faros AI, The AI Engineering Report 2026, source of both review-time figures
• Vibe Coding: Practice, Performance, Productivity, and Risk, the review carrying the +26%, the 19% and the conjecture
• The Impact of AI Coding Assistants on Software Engineering: A Longitudinal Study, supervisory engineering work

Code and quality

• GitClear, The Maintainability Gap
• Debt Behind the AI Boom, 302.6k AI-authored commits
• Characterizing the Quality Profile of AI-Generated C++ in Production
• Safer Builders, Risky Maintainers, the breaking-change rates in both directions

Security

• Veracode, Spring 2026 GenAI Code Security
• Understanding the (In)Security of Vibe-Coded Applications

Benchmarks

• OpenAI, Introducing SWE-bench Verified, the 500-task subset OpenAI built with the original SWE-bench authors
• OpenAI, Why SWE-bench Verified no longer measures frontier coding capabilities, the contamination audit and the decision to stop reporting it
• SWE-bench Pro leaderboard
• Position: Coding Benchmarks Are Misaligned with Agentic Software Engineering

Review, trust and people

• Is Agentic Code Review Helpful?, the CodeRabbit acceptance rates
• Stack Overflow Developer Survey 2025, AI section
• JetBrains, AI Coding Agents: Adoption Trends
• (Im)Paired Programming: Coding Agents Improve Productivity but Harm Understanding
• Canaries in the Coal Mine?, the August 2026 update
• DORA publications
• Andrej Karpathy, Sequoia AI Ascent 2026

Money

• Morgan Stanley on 2026 hyperscaler capex
• Bloomberg, AI Circular Deals