AI on Deterministic Rails

Narrative

The most rigorous empirical work on AI productivity in 2025-2026 comes from METR's randomised controlled trials, which found that experienced open-source developers using early-2025 AI tools took 19% longer to complete tasks than without them. By February 2026, METR had to revise its experimental design because developers were refusing to participate in AI-free conditions, indicating behavioural lock-in even where productivity evidence remained ambiguous. A May 2026 survey of 349 technical workers reported a median 1.4-2x self-reported productivity gain, but METR's own analysis warns that respondents likely overestimate uplift by selecting into tasks where AI helps most. The gap between perceived and measured productivity is a structural feature of the evidence base, not an anomaly.

The reliability literature establishes that orchestration architecture, not raw model capability, is now the binding constraint on production deployment. A February 2026 arXiv paper evaluating 14 agentic models across four reliability dimensions found that recent capability gains had yielded only small improvements in consistency, robustness, and predictability. A companion paper on enterprise agentic architectures documents that production systems increasingly enforce symbolic constraints on tool execution while using LLMs only for high-level decomposition, a hybrid pattern that directly instantiates the AI-plus-deterministic-software thesis. Work on multi-agent orchestration for incident response demonstrated zero quality variance across trials with structured pipelines, compared to single-agent variability, and introduced Decision Quality as a metric that existing LLM benchmarks do not capture. The May 2026 arXiv survey on harness engineering found that evolved harness components transfer cross-model-family with a 12% token reduction on SWE-bench-verified, confirming that the harness encodes general engineering knowledge rather than benchmark-specific tuning.

On open-weight model economics, an arXiv cost-benefit study found that medium-scale models such as Llama-3.3-70B achieve less than 10% accuracy loss versus frontier models at substantially lower total ownership cost, while private inference on consumer Blackwell GPUs reaches cost parity with commercial APIs within one to four months at 30 million tokens per day, then operates at 40-200x lower marginal cost. A November 2025 analysis of token price trends across proprietary and open-weight providers isolated algorithmic efficiency gains from hardware effects, finding rapid price-performance improvement particularly in open-weight inference. The May 2026 paper on token economics for LLM agents formalised the unpredictability of agentic cost: iterative multi-agent loops make total spend non-linear relative to per-token prices, while a concurrent arXiv paper demonstrated that tokenisation ambiguity alone can allow over-reporting of token counts below audit detection thresholds.

The benchmark literature reveals an important methodological tension. SWE-bench Pro, a contamination-resistant extension of SWE-bench Verified, placed even frontier models below 25-45% Pass@1 on long-horizon software engineering tasks, while METR's HCAST provides human-time calibration showing Claude 3.7 Sonnet achieves roughly 50% success on tasks that take a human about one hour. Analysts have noted that METR's evaluation harness is fixed and considerably less capable than production harnesses such as Claude Code, which means benchmark time-horizon figures likely understate what optimised deployments can achieve. The SWE Atlas paper traces Claude Opus from August 2025 to April 2026, showing steady improvement, and quantifies that native scaffolds make substantially more tool calls than minimal scaffolds on identical underlying models, making scaffold choice a primary determinant of benchmark score.