DORA Metrics in the AI Era

Narrative

Academic and technical-preprint literature on DORA-style delivery metrics under AI assistance splits into three empirical strands: controlled productivity experiments, code-quality telemetry studies, and DORA's own large-sample survey research, which functions as a quasi-academic corpus even though it is industry-run.

On controlled experiments, results are genuinely mixed and sample sizes vary enormously. Sida Peng and colleagues' original GitHub Copilot RCT (arXiv 2302.06590, 2023) found that developers with Copilot access completed a scripted HTTP-server task 55.8% faster than controls, a widely cited but narrow, single-task result. Cui, Demirer, Jaffe, Musolff, Peng and Salz scaled this up with three field experiments at Microsoft, Accenture and a Fortune 100 firm, pooling 4,867 developers to find a 26.08% increase (SE 10.3%) in completed tasks under AI access, with larger gains for less experienced developers, published via SSRN and later Management Science. Against this, METR's 2025 RCT (arXiv 2507.09089, Becker, Rush, Barnes and Rein) used 16 experienced open-source developers completing 246 real tasks in codebases they knew well, and found that when AI tools were allowed, developers took 19% longer, despite forecasting a 24% speedup beforehand. A related 2026 METR-adjacent study by Shen and Tamkin (arXiv 2601.20245, Anthropic Fellows Program) ran randomized experiments on library-learning tasks and found AI use impaired conceptual understanding, code reading and debugging without significant average efficiency gains, though participants who fully delegated coding did see some speed gains at the cost of skill formation.

METR's benchmark infrastructure, HCAST (189 tasks across ML, cybersecurity, software engineering and reasoning, calibrated against contracted human baseliners) and RE-Bench (7 open-ended ML research-engineering environments, each roughly 8 human-expert hours), underpins its "time-horizon" capability metric, tracking how long a task an AI agent can reliably complete versus a human. These are methodologically distinct from the developer-productivity RCTs: they measure autonomous agent capability rather than human-plus-AI throughput, and multiple follow-on papers (BRIDGE, Messier) now reuse the HCAST/SWAA/RE-Bench corpus for cross-benchmark calibration work, indicating the dataset has become a de facto standard for this class of evaluation.

On code-quality telemetry, GitClear's commit-level analyses (153 million lines through 2023, then 211 million lines through 2024, then extending into 2026) are vendor research rather than peer-reviewed studies, but are widely cited as corroborating evidence: churn rose from roughly 3.3% pre-AI to 5.7% in 2024 and higher in 2025, copy/pasted code overtook "moved" (refactored) code for the first time in 2024, and duplicated code blocks rose roughly eightfold that year. Independent arXiv work is starting to test these claims directly: "Echoes of AI" (arXiv 2507.00788) runs what its authors describe as the first RCT explicitly targeting AI-assisted code maintainability, and "Intuition to Evidence" (arXiv 2509.19708) and an enterprise RCT on development speed (arXiv 2410.12944) attempt to reconcile self-reported and telemetry-based productivity signals. The SPACE framework (Forsgren, Storey, Maddila, Zimmermann, Houck and Butler, ACM Queue, February 2021) remains the reference multidimensional productivity model that later AI-era papers, including a 2025 systematic review of 39 AI-productivity studies cited in arXiv 2605.01160, use to check which dimensions (typically Activity and Efficiency) are actually measured and which (Communication, Well-being) are neglected.

DORA's own reports supply the largest-sample data points bridging survey and telemetry measurement. The 2024 Accelerate State of DevOps report, drawing on roughly 39,000 respondents including about 3,000 answering AI-specific questions, found a 25% increase in AI adoption was associated with a 7.2% reduction in delivery stability and a 1.5% reduction in delivery throughput, alongside a 2.1% increase in individual productivity, while also reclassifying failed deployment recovery time as a throughput measure and introducing rework rate as a stability measure. The 2025 State of AI-assisted Software Development report and its companion AI Capabilities Model, based on close to 5,000 respondents and over 100 hours of qualitative data, reframe AI as an amplifier of existing organizational conditions and specify seven capabilities (clear AI stance, healthy data ecosystems, AI-accessible internal data, strong version control, working in small batches, user-centric focus, quality internal platforms) that determine whether AI adoption helps or harms the four keys and rework rate.