Software Project Outcomes Over Time, and Is AI Improving Them

Narrative

The academic record on IT project outcomes converges on one durable finding and diverges sharply on everything else. The durable finding is that cost and schedule overruns on software and IT projects are fat-tailed rather than normally distributed. Flyvbjerg, Budzier, Lee, Keil, Lunn and Bester's 2022 Journal of Management Information Systems paper, analysing 5,392 IT projects, established that cost overruns follow a power-law distribution with a large mass of modest overruns and a small number of extreme blowouts, and proposed an interdependency-among-components mechanism to explain it. Their 2026 follow-up in the Journal of Cost Analysis and Parametrics tested 23 project types and found IT stands out with a 73-percentage-point gap between mean and median cost overrun, a signature of extreme fat tails, confirming IT projects carry unusually concentrated tail risk compared with other project categories. Companion arXiv working papers by Flyvbjerg and Budzier quantify this as roughly one in six large IT projects becoming a "black swan" with average cost overruns near 200% and schedule overruns of about 70%, a figure that underlies the widely cited McKinsey-Oxford 2012 study of 5,400-plus IT projects showing average overruns of 45% on cost and 7% on time while delivering 56% less value than predicted.

The Standish CHAOS figures, the most quoted baseline in practitioner literature, are treated far more skeptically in the peer-reviewed record. Eveleens and Verhoef's IEEE Software paper "The Rise and Fall of the Chaos Report Figures" argues Standish's success/challenged definitions rest solely on estimation accuracy, are one-sided, and are "meaningless for benchmarking" once real-world forecasting bias is accounted for. A 1304.4525 arXiv paper on public-sector IT project risk notes that academic scrutiny "debunked" the alarmist Standish-style findings after independent portfolio surveys failed to replicate an IT project crisis of that magnitude, and flags that using Standish's tripartite definitions can actively harm organisations that steer by them. Public-sector failure research (Springer meta-syntheses, case studies from Pakistan and New Zealand hospital IT) consistently finds governance, political stakeholders and underfunded "starved projects" (Flyvbjerg and Budzier document public-sector projects suffering average budget cuts of 75%) as failure modes distinct from, but compounding, the private-sector cost-overrun pattern.

On methodology, systematic literature reviews (Noteboom et al. 2021 HICSS; a 2025 Information and Software Technology SLR covering 208 articles from 1999-2024; a November 2025 arXiv PRISMA review of agile-to-hybrid evolution) repeatedly note that empirical evidence for agile improving project success is thin relative to its adoption, with one 2020 review explicitly stating there are "very few large-scale, empirical studies to support the contention that Agile methods can improve the likelihood of project success" despite widespread practitioner belief. Low-code/no-code research is similarly immature: a 2024 Journal of Systems and Software systematic review of 40 primary studies (2017-2023) and a 2026 ScienceDirect review of LCNC viability both find fragmented, largely qualitative evidence on adoption benefits and governance risk rather than head-to-head bespoke-versus-LCNC outcome data.

The most consequential recent academic development is METR's controlled experimental work on AI-assisted coding. Its July 2025 RCT (Becker, Rush, Barnes and Rein, arXiv 2507.09089), involving 16 experienced open-source developers completing 246 real tasks in mature repositories, found that allowing AI tools (chiefly Cursor Pro with Claude 3.5/3.7 Sonnet) made developers 19% slower, even though developers had forecast a 24% speedup and, after the fact, still believed they had been roughly 20% faster. METR has since revised its experimental design (task-level versus developer-level randomisation) and, per a February 2026 blog update, reports its second round of the same methodology on 57 developers now finds an 18% speedup, using late-2025 tools, illustrating how fast the underlying capability and hence the effect size is moving. METR's benchmark suite (HCAST, RE-Bench and SWAA, combined as METR-HRS) underpins its separate "time horizon" capability metric, showing autonomous task-completion length has doubled roughly every seven months since 2019, a capability trend distinct from, but frequently conflated with, delivery-outcome improvement. Independent longitudinal code-quality evidence from GitClear, based on analysis of over 200 million changed lines of code, documents a parallel maintainability cost: copy-pasted code overtook refactored ("moved") code for the first time in 2024, and code churn roughly doubled from a pre-AI baseline. Google's DORA 2025 State of AI-assisted Software Development report, drawing on nearly 5,000 respondents, frames AI as an "amplifier" that magnifies both high-performing and dysfunctional organisational patterns rather than improving delivery outcomes uniformly, a finding an arXiv 2026 paper on "the productivity-reliability paradox" tries to reconcile against GitHub's own RCT of code readability gains and DORA's simultaneous quality-gain/stability-loss result.