To measure AI coding tools ROI, compare delivery and quality numbers against a baseline you took before the rollout, and put what you pay for seats and usage in the denominator. Don't use developer sentiment or Copilot's acceptance rate as the answer. Both are easy to collect and both can mislead you. The plan below needs data you already have in Git and CI, one staggered rollout, and a spreadsheet. It doesn't need an analytics platform. Our judgment, not a published standard: four to eight weeks of baseline data is enough. The wider Engineering management guides for CTOs and VPs of Engineering cover the rest of the VP's toolkit.
Why surveys and vendor dashboards can't answer the ROI question
Developers are bad judges of their own speed with AI, and the best evidence is METR's 2025 study. Sixteen experienced developers worked through 246 issues. They expected a 24% speedup. With AI they "take 19% longer to complete issues", and afterwards they "still believed AI had sped them up by 20%". METR is careful about scope: "We do not claim that our developers or repositories represent a majority or plurality of software development work." For the full set of speed studies, see what the speed studies show for AI-assisted developers. The point here is narrower: the people being measured were wrong about the direction of their own result.
METR's February 2026 update matters more for anyone planning a measurement. Its follow-up estimated returning developers at 18% slower (confidence interval from 38% slower to 9% faster) and new developers at 4% slower (15% slower to 9% faster). Both intervals include zero. METR itself called the new data "an unreliable signal of the current productivity effect of AI tools" and the central estimate "likely a bad proxy for the real productivity impact", because of selection effects. It reported that between 30% and 50% of developers said they were skipping some tasks because they didn't want to do them without AI. METR describes this as a selection effect that makes its data "only very weak evidence". Judgment: if METR's controlled design is hard to keep clean, a homegrown before/after comparison needs the same caution, which is why step 4 uses cohorts.
Most organizations aren't using a rigorous alternative yet (31% or fewer per metric). LeadDev's Engineering Leadership Report 2026 says "Employee feedback is the most commonly used metric for assessing AI's impact on productivity (63%), but it is an imprecise instrument." Development time per feature (31%), change failure and pull request reversion rates (22%), weekly time saved per developer (22%), and time spent reviewing and updating AI-suggested code (21%) "are used by only around a fifth of organizations." Judgment: the measures that hold up better than self-report are the ones fewer than a third of organizations use.
What vendor metrics tell you, and what they leave out
GitHub's Copilot metrics documentation defines what each number measures. The right-hand column below is judgment, not GitHub's wording.
| Metric | What GitHub says it measures | What it can't tell you |
|---|---|---|
| Acceptance rate | How often developers accept Copilot's suggestions | Whether the accepted code was correct, survived review, or was rewritten a week later |
| Lines suggested, added or deleted | A directional view of Copilot's output in the editor | Whether the lines were needed. More code is a cost as often as a gain |
| Active users | Who used Copilot | Whether use changed what shipped |
| PR lifecycle | Pull request creation and merge counts, median time to merge | Why a number moved. It's the closest thing to a delivery metric in the set |
The documentation also lists limits. The data doesn't include activity from other Copilot surfaces, Copilot Chat on GitHub.com and GitHub Mobile among them, and users need telemetry enabled in their IDE. So the dashboard can undercount, because it leaves out other Copilot surfaces and depends on IDE telemetry.
How to measure AI coding tools ROI in five steps
1. Take the baseline before widening the rollout
Pull four to eight weeks of existing pull request and deployment data for the teams that will get seats, and for at least one comparable team that won't get them yet. Our judgment, not a published standard: that window and that holdout team are enough to start. Freeze the numbers in a sheet with the date. Everything later is compared to this snapshot, which is what keeps the argument about data instead of memory.
If seats are already everywhere, take the baseline from the last period before they landed, and accept that it's rougher.
2. Pick four delivery numbers and one time-per-feature figure
Use the four DORA measures: deployment frequency, lead time for changes, change failure rate and time to restore. DORA metrics and their definitions are covered in their own guide, so this plan doesn't redefine them. Add one figure LeadDev found fewer than a third of organizations use: development time per feature (31%), measured from first commit to merge on a consistent size of work.
Track both sides because DORA's 2025 report, built on "survey responses from nearly 5,000 technology professionals from around the world", found a split. "Unlike last year, we observe a positive relationship between AI adoption on both software delivery throughput and product performance." And: "AI adoption does continue to have a negative relationship with software delivery stability." A throughput-only dashboard would show you the good half.
3. Add review load and rework
Faster code creation can pile up downstream (our inference). DORA's explanation: "AI accelerates software development, but that acceleration can expose weaknesses downstream." Without strong automated testing, mature version control practices and fast feedback loops, DORA says, an increase in change volume leads to instability.
So add two numbers. The first is median time to merge, which comes straight from the Copilot PR lifecycle metrics. The second is rework: reverts, plus changes to the same files within a window you choose. Pick the window (14 days works for many teams) and don't change it mid-quarter. That window is a team choice, not an industry standard.
4. Compare cohorts, not just before and after
Before-and-after alone gets confused by releases, holidays and hiring. Roll seats out in waves, two or three teams at a time, and compare each wave against the teams still waiting. Our judgment, not a published standard: wave size and judging each team against its own baseline rather than the org average.
Expect uneven results, and don't read them as noise. DORA's finding is that "AI doesn't fix a team; it amplifies what's already there. Strong teams use AI to become even better and more efficient. Struggling teams will find that AI only highlights and intensifies their existing problems." If one team's instability rises while another's holds, treat that as a result worth acting on. It tells you where to fix testing and review before buying more seats.
5. Put cost in the denominator
Total program cost is seats plus any usage-based charges, taken from the invoice rather than from list prices. Read Copilot's new billing rules from 1 October 2026 (upfront seat charging for card and PayPal customers), which are already in effect, so your quarterly cost reflects the plan you're actually on. This article doesn't invent a typical total cost, and neither should your model.
The formula: (value of measured gain minus total program cost) divided by total program cost.
The hard part is "value of measured gain". Our recommendation, as judgment: count a saved hour only at the rate that it turns into shipped work. If merge times fall by a day but the team's deployment frequency doesn't move, you've saved nothing the CFO can see. Haircut hours saved by the share you can trace to shipped output, and say so in the model.
Pulling the numbers without buying a platform
Copilot's usage data is available through GitHub's Copilot usage metrics REST API. The org-level report endpoint is GET /orgs/{org}/copilot/metrics/reports/organization-28-day/latest. There are also organization-1-day, users-1-day, users-28-day, repos-1-day and user-teams-1-day reports. Responses return download_links you fetch to get the data. A classic token needs the read:org scope. A fine-grained token needs the "View Organization Copilot Metrics" permission. Check the page for current paths before you build on them. The documentation says reports start from 10 October 2025, history reaches back up to one year, and a day's data appears within two full UTC days.
Everything else in the plan comes from data you own: merge and deploy timestamps from your Git host and CI, reverts from commit history, incident times from your pager. Our judgment: one engineer can wire a weekly export into a sheet in a couple of days. Buy a platform later if the sheet earns it.
What this means for your AI tooling budget
You're deciding whether to keep, widen or cut AI seat spend, and what to tell the board in place of an acceptance rate. Report three things: delivery and instability against the baseline, review load, and cost per seat. Label survey results as a sentiment signal. METR's perception gap and LeadDev's finding that most organizations lean on employee feedback are the reasons.
Before you widen seats, set a review date and a stop rule. For example: if change failure rate rises for two consecutive months in a cohort and throughput hasn't, pause that cohort. This is judgment, not a benchmark, but a rule written in advance beats a debate held after the invoice.
For the cost side, AI coding tool seat costs set against a senior hire is the next stop. If the numbers point at team shape instead of tooling, read how many juniors and seniors a startup should hire in the AI era.
FAQ
How do you measure ROI of AI coding tools?
Take a baseline of delivery data before rollout, roll seats out in waves, and compare cohorts on DORA throughput and stability, review time and rework. Then apply (value of measured gain minus total program cost) divided by total program cost, with hours saved counted only as far as they become shipped work.
Is acceptance rate a good productivity metric?
No. GitHub defines it as how often developers accept Copilot's suggestions. It says nothing about whether the code was right, survived review, or was needed. Treat it as a usage signal.
Do DORA metrics work for AI-assisted teams?
Yes, if you track both halves. DORA's 2025 data showed a positive relationship between AI adoption and delivery throughput and a continued negative one with delivery stability. Baseline all four metrics and read throughput and instability together.
Why did my developers feel faster but delivery didn't move?
Self-report and measured output diverge. In METR's 2025 study developers believed AI had sped them up by 20% while they took 19% longer. DORA also points to downstream bottlenecks: more change volume without strong testing and fast feedback loops leads to instability. Check review time and rework before blaming the tool.