Developer & Tech
Measuring Whether AI Actually Made Your Team Faster
By Jim Vernon, Editor, AI Intelligence International · Published 1 April 2026 · Reviewed against our editorial standards · About the author
Vendor metrics report suggestions accepted and lines generated. Neither has a defensible relationship to whether software reached users faster or more reliably, which is the only thing the investment can be judged on.
This article covers what to measure instead, over what period, and what a realistic result looks like.
Key takeaways
- Why acceptance rate is misleading: Accepting a suggestion and then rewriting it counts as acceptance.
- Measure delivery, not activity: The four widely used delivery metrics — change lead time, deployment frequency, change failure rate and time to restore — are the right frame, because they capture both speed and the quality cost of speed.
- Where the time actually goes: Coding is typically a minority of a developer's week.
- Review load is the hidden cost: More code produced means more code to review, and review capacity does not scale with generation.
Why acceptance rate is misleading
Accepting a suggestion and then rewriting it counts as acceptance. So does accepting a suggestion that later causes an incident.
The metric is measuring the tool's fit with your typing, not its contribution to output. It is useful to the vendor and close to useless to you.
Lines generated is worse, because more code is a cost. A team producing the same functionality in less code has improved, and the metric records a decline.
Measure delivery, not activity
The four widely used delivery metrics — change lead time, deployment frequency, change failure rate and time to restore — are the right frame, because they capture both speed and the quality cost of speed.
Speed metrics alone will improve with assistance almost automatically. Whether the failure metrics hold is the actual question.
Track all four together or none. Improvement in two while the other two degrade is a common and easily hidden outcome.
Where the time actually goes
Coding is typically a minority of a developer's week. If assistance halves coding time and coding is a quarter of the week, the ceiling on total improvement is around twelve per cent.
Measure the split before drawing conclusions. Teams frequently discover the bottleneck is review latency, environment problems or requirement churn, none of which generation addresses.
Where review is the bottleneck, adding generation capacity makes it worse by increasing the volume awaiting review.
Review load is the hidden cost
More code produced means more code to review, and review capacity does not scale with generation. This shows up as growing pull request queues rather than as a metric anyone is watching.
Track time-to-first-review and pull request size distribution. Both degrade before anything else becomes visible.
If pull request size is rising, the tool is shifting work to reviewers rather than removing it.
Time horizon matters
Measure over at least a quarter. Novelty effects run for several weeks, and defects introduced in month one surface in month three.
Compare like periods. A quarter containing a major incident or a reorganisation tells you about the incident.
Keep a baseline from before adoption. Teams that start measuring after adopting cannot answer the question at all.
Qualitative signals worth collecting
Ask developers two questions monthly: how much of your week was spent on work you would call valuable, and how confident are you in code you did not write.
The second question is a leading indicator of comprehension debt and moves earlier than any delivery metric.
Take low scores seriously even when the delivery numbers look good. Comprehension problems surface as incidents months later.
Worked example: a quarter of measurement
Team of nine, baseline captured for the quarter before rollout: median change lead time 4.1 days, deploys 12 per week, change failure rate 14%, median time to restore 90 minutes.
Quarter after: lead time 3.3 days, deploys 17 per week, change failure rate 21%, time to restore 145 minutes. Speed improved meaningfully and reliability degraded meaningfully.
Investigation found pull request size had risen by roughly sixty per cent and time-to-first-review had doubled. Reviewers were approving larger diffs with the same attention budget.
The response was a hard pull request size limit and a review rotation, not a change to the tooling. The following quarter held the speed gain with failure rate back at 15%, which is the outcome the original numbers would have missed entirely.
Why line counts and acceptance rates mislead
Suggestion acceptance rate measures how agreeable the completions are, not whether the software got better. It rises when developers accept boilerplate and falls when they are doing the hardest, most valuable work of the quarter.
Lines of code is worse: assistants generate verbose code readily, and a productivity metric that rewards volume produces exactly the codebase you would expect.
Both numbers are fine as diagnostics — a collapse in acceptance rate says something changed — and unusable as targets, because they are trivially gamed by working badly.
Measure delivery and stability instead
Lead time from first commit to production, deployment frequency, change failure rate, and time to restore describe whether the team actually ships working software. They are hard to game without genuinely improving, and they existed before AI, so you can compare across the adoption boundary.
Pair them with a quarterly developer survey question: where does the tooling help, and where does it get in the way. Two consistent themes from a survey usually beat a dashboard for deciding what to change next.
Establish the baseline before rollout. Teams that start measuring after adoption have no way to separate the tool's effect from a quieter quarter or a new hire finding their feet.
Frequently asked questions
What is a realistic productivity gain?
Published studies and internal measurements cluster in the range of a few per cent to around twenty per cent on delivery throughput, with wide variance by task type. Claims of multiples generally measure coding time in isolation rather than delivery.
Should individual developers be measured?
No. Individual metrics are gameable, demoralising and poorly correlated with contribution. Measure at team level where the delivery outcome actually exists.
How long before results are trustworthy?
One full quarter minimum, ideally two, with a pre-adoption baseline. Shorter windows measure enthusiasm.
What if the numbers get worse?
Usually the bottleneck was not coding. Look at review latency, pull request size and requirement churn before concluding the tool is at fault.
Should individual AI usage be tracked?
No. Individual tracking produces performative usage and damages trust; measure team-level delivery outcomes instead.