5 November 2026

·

4 min read

Delivery & CI/CDmeasure developer productivity with AI toolsvalidation in the age of AI

Measuring AI Uplift on Your Own Team: Why METR Changed Its Experiment Design

METR could not produce a dependable estimate of AI-assisted developer productivity from its late-2025 experiment. The reasons it gives tell you how to measure uplift on your own team, and why fixed multipliers do not belong in plans.

Anystack Engineering

Validation usually points at the code an agent wrote. It should also point at the claims made about AI itself. The most common claim in a planning deck right now is a productivity multiplier, and the research record on that number is thinner than the confidence attached to it.

On 24 February 2026 METR published an update on its attempt to estimate AI-assisted developer productivity. The post is a research-programme update, not a peer-reviewed paper, and it reports a negative result about its own method: selection effects and unreliable time measurement prevented its late-2025 experiment from producing a dependable estimate. METR designed a study to measure AI uplift, found the measurement would not hold, and said so.

That admission is more useful to an engineering leader than any single headline figure, because it names the two things that make uplift hard to measure and shows what a credible measurement has to control for.

What METR reported

Two problems broke the estimate.

The first is selection effects. Developers who adopt AI tools, and tasks chosen for AI assistance, are not a random sample of your work. If faster developers reach for the tool first, or if easy tasks get routed to it, a before-and-after comparison measures the selection, not the tool.

The second is unreliable time measurement. Self-reported or loosely instrumented timing drifts. When the effect you are chasing is smaller than the noise in how you record it, the result is a guess with a decimal point.

METR stops where the evidence stops. It does not publish a corrected multiplier. The takeaway is about method, not about a number.

Why fixed multipliers do not belong in plans

A fixed multiplier in a capacity plan assumes the uplift is a property of the tool. The METR update shows it is a property of the tool, the task, the person and the measurement together. Move any of those and the figure moves.

There is a second cost that vendor figures tend to leave out: review and rework. A model can produce a diff quickly and still move work downstream, into the reviewer's queue and into the next sprint when the change needs reopening. A measurement that times only the first draft of the code counts the fast part and ignores the slow part. If your plan credits the fast part and your delivery pays the slow part, the plan is wrong in a predictable direction.

Three actions for this week

Measure on the work and the team using the tool, not on a borrowed figure.

  • Define the unit you will measure before you measure it. Pick a repeatable task type, for example a CRUD endpoint with tests or a bug fix with a reproduction, and measure cycle time for that type with and without assistance. Comparing like with like is the control METR lost.
  • Count review and rework inside the same window. Record time to merge and whether the change reopened within a fixed period. A gain in authoring time that is lost to longer reviews or a reopened change is not a gain, and you will only see it if the window extends past the first commit.
  • Instrument time from the system, not from memory. Use timestamps your pipeline already records: pull request opened, first review, merged, reverted. Self-reported duration is the measurement METR called unreliable. Timestamps you did not have to ask anyone to record remove that source of drift.

None of this requires a research team. It requires deciding the unit, holding it steady, and reading timestamps you already have. Run it for a few weeks across the people and tasks using AI assistance, and you will have an estimate grounded in your own repositories rather than someone else's benchmark.

What a credible estimate looks like

Treat the output as a range on a specific task type for a specific team, with review and rework inside the measured window, not as a company-wide constant. If a sceptical colleague cannot see which tasks, which people and which timestamps produced the figure, it is not ready to go in a plan.

A simpler version is a delivery meter: a small set of per-change measurements kept consistently over time, so the trend is visible and any claimed effect has to show up in it. The meter does not tell you a multiplier. It tells you whether anything changed, a question most plans skip.

How Anystack helps teams act on this

An Anystack pod works inside your repositories and pipelines, and every change lands on a delivery meter that the client keeps at handover. Each change starts as a written spec, passes machine checks before human review, and carries a named reviewer of record, so review and rework are visible on the same record as the change rather than lost between tools. AI assists the engineers and never occupies an engineer slot. The point is to give you per-change measurements you can read yourself, on your own work, after we leave.


*Anystack Engineering runs three-person engineering pods that deliver inside your repositories with a written spec for every change, a named reviewer of record and delivery measured per engagement. Engagements from £18,000 per month. How the pod delivers.*

Discussions

0

Technical analysis, failure mode challenges, and reproduction observations.

Add to discussion

0/3000

No discussions yet. Share an observation or technical question above.

Start a conversation

Share the engineering context and delivery objective when you are ready to discuss the work.

Contact Anystack →

See the evidence

Read selected engineering work and its provenance.

Browse selected work →