Token Cost of Ownership

Narrative

{
  "lane": "academic",
  "label": "Academic & arXiv",
  "sources": [
    {
      "title": "The Rising Costs of Training Frontier AI Models",
      "url": "https://arxiv.org/html/2405.21015v2",
      "date": "2024-05",
      "outlet": "arXiv / Epoch AI Research",
      "significance": "Foundational Epoch AI paper establishing that frontier model training costs have grown ~2.4× per year since 2016, with the largest runs projected to exceed $1 billion by 2027 — providing the empirical baseline for all academic estimates of training-cost amortisation into per-token prices."
    },
    {
      "title": "RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents Against Human Experts",
      "url": "https://arxiv.org/abs/2411.15114",
      "date": "2024-11",
      "outlet": "arXiv / METR",
      "significance": "METR's benchmark measuring AI agent performance on ML research engineering tasks against human experts, directly relevant to whether AI can self-accelerate the inference efficiency improvements that drive cost reduction curves."
    },
    {
      "title": "HCAST: Human-Calibrated Autonomy Software Tasks",
      "url": "https://metr.org/hcast.pdf",
      "date": "2024",
      "outlet": "METR",
      "significance": "Defines METR's 189-task benchmark suite calibrated to human completion times, enabling cost-per-unit-of-capability comparisons across model generations — critical for evaluating whether API prices reflect capability-adjusted subsidies."
    },
    {
      "title": "Measuring AI Ability to Complete Long Software Tasks",
      "url": "https://arxiv.org/abs/2503.14499",
      "date": "2025-03",
      "outlet": "arXiv / METR",
      "significance": "Documents METR's finding that AI time horizons doubled roughly every 7 months through early 2025, providing the capability growth denominator needed to compute cost-per-useful-work trajectories alongside per-token price declines."
    },
    {
      "title": "The Price of Progress: Algorithmic Efficiency and the Falling Cost of AI Inference",
      "url": "https://arxiv.org/html/2511.23455v1",
      "date": "2025-11",
      "outlet": "arXiv / Epoch AI Research",
      "significance": "Constructs the largest dataset of AI benchmark prices using Epoch AI's Benchmark Hub, finding ~10× per year cost reduction for a given performance level with variance ranging from 9× to 900× per year by benchmark type — the definitive quantification of the gap between API prices and cost trajectories."
    },
    {
      "title": "Beyond Benchmarks: The Economics of AI Inference",
      "url": "https://arxiv.org/html/2510.26136v1",
      "date": "2025-10",
      "outlet": "arXiv",
      "significance": "Analyses the cost-efficiency frontier across model families, showing that MoE architectures and quantization deliver frontier-quality output at 3–5× lower cost than dense models — a key mechanism behind the accelerating inference price declines observed since 2024."
    },
    {
      "title": "Inference Economics of Language Models",
      "url": "https://arxiv.org/html/2506.04645v1",
      "date": "2025-06",
      "outlet": "arXiv",
      "significance": "First
```json
{
  "lane": "academic",
  "label": "Academic & arXiv",
  "sources": [
    {
      "title": "The Rising Costs of Training Frontier AI Models",
      "url": "https://arxiv.org/html/2405.21015v2",
      "date": "2024-05",
      "outlet": "arXiv / Epoch AI Research",
      "significance": "Foundational Epoch AI paper establishing frontier training costs growing ~2.4× per year since 2016, projecting runs above $1 billion by 2027 and grounding all academic estimates of training-cost amortisation into per-token pricing."
    },
    {
      "title": "RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents Against Human Experts",
      "url": "https://arxiv.org/abs/2411.15114",
      "date": "2024-11",
      "outlet": "arXiv / METR",
      "significance": "METR benchmark measuring AI on ML research engineering tasks against human experts — directly relevant to whether AI can self-accelerate the inference efficiency improvements driving cost reduction."
    },
    {
      "title": "HCAST: Human-Calibrated Autonomy Software Tasks",
      "url": "https://metr.org/hcast.pdf",
      "date": "2024",
      "outlet": "METR",
      "significance": "Defines METR's 189-task benchmark suite calibrated to human completion times, enabling cost-per-unit-of-capability comparisons across model generations that reveal capability-adjusted subsidy levels in API pricing."
    },
    {
      "title": "Measuring AI Ability to Complete Long Software Tasks",
      "url": "https://arxiv.org/abs/2503.14499",
      "date": "2025-03",
      "outlet": "arXiv / METR",
      "significance": "Documents that AI time horizons doubled roughly every 7 months through early 2025, providing the capability growth denominator for computing cost-per-useful-work trajectories alongside per-token price declines."
    },
    {
      "title": "The Price of Progress: Algorithmic Efficiency and the Falling Cost of AI Inference",
      "url": "https://arxiv.org/html/2511.23455v1",
      "date": "2025-11",
      "outlet": "arXiv / Epoch AI Research",
      "significance": "Largest-to-date dataset of AI benchmark prices, finding ~10× per year cost reduction for a given performance level with variance from 9× to 900× per year by benchmark type — the definitive academic quantification of the API-price-to-cost-trajectory gap."
    },
    {
      "title": "Beyond Benchmarks: The Economics of AI Inference",
      "url": "https://arxiv.org/html/2510.26136v1",
      "date": "2025-10",
      "outlet": "arXiv",
      "significance": "Analyses the cost-efficiency frontier across model families, documenting that MoE architectures and quantization deliver frontier-quality output at 3–5× lower per-token compute cost than dense models."
    },
    {
      "title": "Inference Economics of Language Models",
      "url": "https://arxiv.org/html/2506.04645v1",
      "date": "2025-06",
      "outlet": "arXiv",
      "significance": "First formal economic model of the cost-per-token vs. generation-speed trade-off under compute, memory bandwidth, and network latency constraints — the theoretical backbone for data centre TCO and LLM pricing strategy analysis."
    },
    {
      "title": "An Inquiry into Datacenter TCO for LLM Inference with FP8",
      "url": "https://arxiv.org/html/2502.01070v4",
      "date": "2025-02",
      "outlet": "arXiv",
      "significance": "Most rigorous public framework for data centre total-cost-of-ownership at FP8 precision, benchmarking AI accelerators across operational requirements and decomposing hardware, energy, and networking cost shares per token."
    },
    {
      "title": "A Cost-Benefit Analysis of On-Premise Large Language Model Deployment: Breaking Even with Commercial LLM Services",
      "url": "https://arxiv.org/html/2509.18101v3",
      "date": "2025-09",
      "outlet": "arXiv",
      "significance": "Shows on-premise inference can break even against commercial API pricing in 0.3–3 months for moderate workloads, empirically bounding how far below true serving cost commercial API prices are currently set."
    },
    {
      "title": "From Prompts to Power: Measuring the Energy Footprint of LLM Inference",
      "url": "https://arxiv.org/html/2511.05597",
      "date": "2025-11",
      "outlet": "arXiv",
      "significance": "Large-scale study across 32,500+ measurements on 21 GPU configurations and 155 model architectures, directly translating inference workloads to energy cost and linking power consumption to per-token pricing floors."
    },
    {
      "title": "Energy Considerations of Large Language Model Inference and Efficiency Optimizations",
      "url": "https://aclanthology.org/2025.acl-long.1563.pdf",
      "date": "2025",
      "outlet": "ACL 2025",
      "significance": "Peer-reviewed ACL paper establishing that inference accounts for >90% of LLM lifecycle energy consumption, formally inverting the conventional framing that training dominates AI energy cost and therefore long-run token pricing."
    },
    {
      "title": "TokenPowerBench: Benchmarking the Power Consumption of LLM Inference",
      "url": "https://arxiv.org/html/2512.03024v1",
      "date": "2025-12",
      "outlet": "arXiv",
      "significance": "First dedicated power-consumption benchmark measuring joules per token across batch sizes, context lengths, and quantization levels — establishing the empirical methodology for computing true electricity cost per token."
    },
    {
      "title": "Concentrated Siting of AI Data Centers Drives Regional Power-System Stress Under Rising Global Compute Demand",
      "url": "https://arxiv.org/html/2604.06198v1",
      "date": "2026-04",
      "outlet": "arXiv",
      "significance": "April 2026 paper showing geographic clustering of AI data centres creates nonlinear regional grid stress and capacity market price spikes — a structural cost input absent from standard per-token pricing models."
    },
    {
      "title": "Electricity Demand and Grid Impacts of AI Data Centers: Challenges and Prospects",
      "url": "https://arxiv.org/html/2509.07218v4",
      "date": "2025-09",
      "outlet": "arXiv",
      "significance": "Projects AI data centre electricity demand reaching 9.1–11.7% of total US energy by 2030, quantifying the energy supply constraints that represent the most durable long-run cost floor for inference pricing."
    },
    {
      "title": "How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference",
      "url": "https://arxiv.org/html/2505.09598v1",
      "date": "2025-05",
      "outlet": "arXiv",
      "significance": "Multi-dimensional resource-cost benchmarks (energy, water, carbon) per token across major LLM families, enabling full TCO comparisons that include cooling and embodied resource costs absent from API pricing."
    },
    {
      "title": "From Efficiency Gains to Rebound Effects: The Problem of Jevons' Paradox in AI's Polarized Environmental Debate",
      "url": "https://arxiv.org/abs/2501.16548",
      "date": "2025-01",
      "outlet": "arXiv / ACM FAccT 2025",
      "significance": "Peer-reviewed FAccT 2025 paper establishing Jevons' Paradox as the principal framework for AI cost-demand dynamics: efficiency gains (cheaper tokens) consistently produce higher aggregate energy consumption, preventing cost floors from declining indefinitely."
    },
    {
      "title": "Will Neural Scaling Laws Activate Jevons' Paradox in AI Labor Markets?",
      "url": "https://arxiv.org/html/2503.05816",
      "date": "2025-03",
      "outlet": "arXiv",
      "significance": "Applies a time-varying elasticity of substitution framework to scaling dynamics, projecting how efficiency-driven token cost reductions translate into demand expansion — directly relevant to whether cost savings are absorbed by usage growth through 2028."
    },
    {
      "title": "The Jevons Paradox in Cloud Computing: A Thermodynamics Perspective",
      "url": "https://arxiv.org/html/2411.11540v1",
      "date": "2024-11",
      "outlet": "arXiv",
      "significance": "Documents historical cloud compute efficiency rebounds as the closest academic precedent for AI token pricing dynamics, including how AWS and GCP priced below cost during market expansion phases before normalisation."
    },
    {
      "title": "Forecasting GPU Performance for Deep Learning Training and Inference",
      "url": "https://arxiv.org/html/2407.13853v3",
      "date": "2024-07",
      "outlet": "arXiv",
      "significance": "Models GPU performance scaling trajectories (H100 through projected Rubin-generation hardware) providing the hardware supply-side baseline for projecting FLOPS-per-dollar improvements and inference cost reductions through 2028."
    },
    {
      "title": "A Survey on Efficient Inference for Large Language Models",
      "url": "https://arxiv.org/pdf/2404.14294",
      "date": "2024-04",
      "outlet": "arXiv / TMLR 2024",
      "significance": "Comprehensive TMLR survey of inference optimisation techniques — quantization, pruning, distillation, KV-cache compression, speculative decoding — documenting the algorithmic efficiency toolbox driving the ~10× annual cost reduction per performance unit."
    },
    {
      "title": "Trends in Frontier AI Model Count: A Forecast to 2028",
      "url": "https://arxiv.org/html/2504.16138",
      "date": "2025-04",
      "outlet": "arXiv",
      "significance": "Projects the number of frontier AI models through 2028, providing the demand-side forecast for inference infrastructure and indicating whether competitive pricing pressure or market consolidation will dominate token cost trajectories."
    },
    {
      "title": "LLM Inference Prices Have Fallen Rapidly but Unequally Across Tasks",
      "url": "https://epoch.ai/data-insights/llm-inference-price-trends",
      "date": "2025",
      "outlet": "Epoch AI Data Insights",
      "significance": "Epoch AI's granular tracking of inference price declines by benchmark type (9×–900× per year variance), the most detailed public dataset on how API price evolution differs between commodity and frontier performance tiers."
    },
    {
      "title": "How Much Does It Cost to Train Frontier AI Models?",
      "url": "https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models/",
      "date": "2024",
      "outlet": "Epoch AI Blog",
      "significance": "Epoch AI's methodology for estimating training costs from observable compute budgets and hardware pricing — the primary public framework for amortising training expenditure across inference tokens to derive a true cost floor."
    },
    {
      "title": "Training Compute Costs Are Doubling Every Eight Months for the Largest AI Models",
      "url": "https://epoch.ai/data-insights/cost-trend-large-scale",
      "date": "2024",
      "outlet": "Epoch AI Data Insights",
      "significance": "Documents the ~2.4×/year growth in frontier training compute costs, establishing the accelerating training cost curve that must be amortised into inference pricing and that constrains the duration of below-cost API subsidisation."
    },
    {
      "title": "Evaluating Frontier AI R&D Capabilities of Language Model Agents Against Human Experts",
      "url": "https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/",
      "date": "2024-11",
      "outlet": "METR Blog",
      "significance": "METR's announcement of RE-Bench results showing frontier AI agents match human ML experts on tasks up to ~2 hours, a capability milestone informing projections of AI-assisted infrastructure optimisation and self-reinforcing cost reduction loops."
    }
  ],
  "narrative": "The academic and arXiv literature from 2023–2026 documents a structural asymmetry at the heart of AI token economics: frontier training costs are rising ~2.4× per year (Cottier et al., Epoch AI, arXiv:2405.21015), while inference cost per unit of AI performance is falling ~10× per year on average (Epoch AI arXiv:2511.23455), with per-benchmark variance of 9× to 900× per year. This divergence implies that published API prices function as strategic subsidies rather than cost-recovery instruments — a thesis empirically supported by arXiv:2509.18101, which finds on-premise inference can break even against commercial APIs in as little as 0.3–3 months, bounding how far below true serving cost API prices are set. The infrastructure economics papers fill in the cost structure: arXiv:2502.01070 provides the most rigorous public TCO framework for FP8 inference; ACL 2025 (aclanthology:2025.acl-long.1563) and arXiv:2512.03024 (TokenPowerBench) establish that inference accounts for over 90% of LLM lifecycle energy; and arXiv:2511.05597 measures actual joules-per-token across 155 model architectures. On the demand side, METR's HCAST, RE-Bench (arXiv:2411.15114), and Long Tasks (arXiv:2503.14499) document AI capability doubling every ~7 months — a rate that drives inference demand growth faster than hardware efficiency gains reduce per-token cost.\n\nThe energy and grid literature represents the most consequential academic contribution for long-run pricing analysis. arXiv:2509.07218 projects AI data centres consuming 9–12% of US electricity by 2030; arXiv:2604.06198 (April 2026) shows geographic clustering creates nonlinear regional grid stress and capacity-market clearing price spikes of 10× or more — structural costs entirely absent from per-token pricing models. The Jevons Paradox papers (arXiv:2501.16548, ACM FAccT 2025; arXiv:2503.05816) provide formal theoretical grounding for why efficiency-driven price declines cannot continue indefinitely: empirical and modelling evidence consistently shows cheaper tokens produce proportionally more demand, absorbing efficiency gains and sustaining energy cost pressure. Taken together, these sources indicate that the cost floor for frontier inference will be set by energy and capital constraints — not by algorithmic efficiency alone — and that the rapid price declines observed between 2023 and 2025 reflect intentional subsidisation strategies more than structural cost normalisation.",

  "model_context": "The academic economics of LLM token pricing sit at an unusual intersection of industrial organisation, energy systems research, and computer architecture that had no established literature before 2022. Unlike cloud compute pricing — which accumulated decades of academic cost-accounting work as hardware commoditised under well-understood Moore's Law dynamics — AI inference pricing involves opaque training-cost amortisation across uncertain demand curves, architectures changing faster than cost models can stabilise (dense transformers to MoE to speculative decoding), and deliberate strategic pricing below cost. Regulatory disclosure requirements for inference cost do not exist, so academic researchers must infer serving costs from observable proxies: disclosed compute budgets in FLOPs, GPU cluster configurations at hyperscale data centres, published energy contracts, and secondary-market hardware pricing. Epoch AI's methodology — tracking hardware depreciation, energy tariffs, and interconnect costs per training run — is the most rigorous publicly available framework, but it applies primarily to training cost rather than inference TCO. The inference pricing literature (arXiv:2502.01070, arXiv:2509.18101) attempts to fill that gap, but remains limited by the absence of lab-level disclosure. The Epoch AI 'Price of Progress' series is therefore the closest academic equivalent to true-cost analysis, and its finding of a 10× annual cost-per-performance decline that exceeds the rate at which API prices have fallen implies that either subsidies are widening, margins are improving, or both simultaneously.\n\nMETR's contribution to the cost-trajectory literature is indirect but structurally important. The 'time horizon' metric — the human-task-completion time at which a model achieves 50% success — creates a capability-denominated unit that, divided into API cost, yields cost-per-unit-of-human-equivalent-work. This framing is more economically meaningful than cost-per-token because it accounts for the fact that models are completing progressively longer and more complex tasks within the same token budget. METR's data suggests time horizons doubled approximately every 7 months through early 2025; combined with ~10× annual per-token cost reductions, the implied cost-per-useful-work is falling faster than any prior productivity technology. This has implications for subsidisation sustainability: if labs compete on cost-per-capability rather than cost-per-token, the subsidy required to maintain competitive position may be shrinking even as nominal infrastructure spend grows. RE-Bench adds a further feedback dimension: if AI agents can accelerate ML research engineering (including inference optimisation), there is a self-reinforcing loop between capability growth and cost reduction that current academic cost models do not incorporate.\n\nThe energy and data centre economics literature has emerged as the most credible academic source of long-run cost floor analysis. The finding from multiple 2025 papers that inference dominates LLM energy consumption — not training — inverts the common framing in media coverage and means that electricity tariff dynamics, not GPU depreciation, will increasingly determine the minimum achievable per-token price. The geographic concentration finding (arXiv:2604.06198, April 2026) is particularly significant: it shows that PJM-style capacity market clearing prices can spike 10× in a single auction cycle driven by AI data centre buildout, a tail risk that per-token pricing models and most industry forecasts have not priced in. Labs that secure long-term power purchase agreements in low-cost grid regions (nuclear-backed Midwest, hydro-backed Pacific Northwest) gain structural cost advantages invisible in hardware-only analyses. The most recent academic work treats these as 'energy cost floors' — bounds below which per-token prices cannot fall regardless of algorithmic improvements — and the April 2026 literature suggests those floors are higher, and closer in time, than the rapid price declines of 2023–2025 implied."
}