AI Economics — Why Inference Now Costs More Than Training

In brief

Training a large language model costs tens to hundreds of millions of dollars — a massive, one-time investment amortized over several years. But in 2024, OpenAI spent approximately $2.3 billion in inference compute alone, roughly fifteen times the estimated training cost of GPT-4.5 [TechCrunch / futuresearch.ai, 2024]. The economics of generative AI have flipped: the upfront cost of building the model is now minor compared to the ongoing cost of running it. This article documents that shift — its causes, its numbers, its impact on margins, and why reasoning models like o3 are amplifying the problem further.

Understanding the distinction: CAPEX and OPEX in AI

Think of a printing press. Buying the machines, building the facility, training the staff: that’s the initial investment — a heavy, bounded expenditure. Then every book printed consumes paper, ink, electricity, and labor: that’s the operational cost, proportional to volume.

Large language models follow the same logic:

Key terms

• CAPEX (Capital Expenditure): capital investment. In AI, this is training cost — GPU rental, data acquisition, R&D staff. One-time expenditure, amortized over several years.
• OPEX (Operational Expenditure): recurring operational cost. In AI, this is inference — the compute consumed by every user request. Proportional to usage volume.
• Training: the phase where the model learns from a massive corpus. Happens once (or in iterative runs). This is the CAPEX.
• Inference: the moment when the model answers a query. Happens billions of times. This is the OPEX.

Through 2022, the debate about AI costs focused almost entirely on training. Makes sense: the numbers were spectacular and models were rarely used at scale. Since 2023-2024, the equation has reversed. The hundreds of millions of users of ChatGPT, Claude, and Gemini generate query volumes that make training costs look minor over a model’s lifetime.

In plain terms: building the model is expensive once. Running it is expensive indefinitely. In 2024, the second category dominates the budgets.

Timeline 2020–2025: what the big models actually cost

GPT-3 (2020): $4–5 million to train

GPT-3, 175 billion parameters, remains the foundational reference point. Three sources converge on a $4–5 million range in cloud-equivalent cost:

• Lambda Labs (2020) estimates $4.6M to reproduce GPT-3 training at GPU cloud rates of the time [cudocompute.com, citing Lambda Labs].
• Fortune (2024) cites a consensus range of $2–4M for OpenAI’s internal compute cost (proprietary hardware, no cloud markup).
• Epoch AI (2022), using a hardware-aware methodology, arrives at $700K–$1.1M in 2020 USD via GPU theoretical compute performance — while noting “true costs could be several times larger” [Epoch AI blog, 2022].

The $10M figure frequently cited in the press is likely inflated [NON VÉRIFIÉ — no primary OpenAI source].

For reference: the GPT-3 inference price at launch was approximately $60 per million tokens in November 2021 [a16z LLMflation, 2024]. This number serves as the baseline for measuring the price collapse that followed.

PaLM and Megatron-Turing (2022): similar ballpark

Google’s PaLM (540 billion parameters) is estimated between $3M and $12M [Epoch AI; cudocompute.com]. NVIDIA and Microsoft’s Megatron-Turing NLG (530 billion parameters) is estimated at approximately $2.1M in 2022 USD [Epoch AI, 2022]. These figures show that even the giants of 2022 were still affordable by comparison to what came next.

GPT-4 (2023): the jump to $63–100 million

GPT-4 marks a structural break in training economics. Three sources triangulate a $63–100M range:

1. SemiAnalysis (Dylan Patel & Gerald Wong, July 2023): GPT-4 architecture + infrastructure + training cost analysis [newsletter.semianalysis.com — partially paywalled].
2. Sam Altman (Wall Street Journal, 2023): direct statement of “$100M+” for GPT-4 training cost [NON VÉRIFIÉ without primary WSJ access].
3. Juma.ai (2024): independent computational estimate → ~$63M assuming H100 cloud rates, 90–100 days of training [juma.ai].

OpenAI never disclosed official numbers. The range is wide, but the orders of magnitude are consistent across independent sources.

At launch (March 2023), GPT-4 inference was priced at $30–60 per million tokens (8K input/output). The price dropped to $10–30 for GPT-4o in May 2024 [OpenAI pricing, 2024].

Llama 3.1 405B (Meta, 2024) and DeepSeek V3 (2024)

Meta officially documents 30.8 million GPU-hours on H100-80GB to train Llama 3.1 405B [Meta AI blog, 2024]. At cloud H100 rates ($2–3/hour/GPU), this represents $61–92M in cloud-equivalent cost. Since Meta uses its own clusters (49,152 H100 GPUs), the actual internal cost is significantly lower.

DeepSeek V3 (December 2024) declared a training compute cost of just $5.58M on 2,048 H800 GPUs over 55 days [DeepSeek technical report, 2024]. This figure is highly contested: it covers only the final training run, not total R&D, preliminary experiments, or the H800 GPUs at Chinese internal pricing [The Register, Sept. 2025; Martin Vechev, INSAIT — The Recursive]. The economic lesson remains valid: DeepSeek’s algorithmic innovations (MoE, MLA, FP8 mixed precision) dramatically reduced FLOP per token. The marketing number should be read with caution.

In plain terms: from GPT-3 in 2020 ($4–5M) to GPT-4 in 2023 ($63–100M), the cost to train a frontier model multiplied by 15–20 in three years. The trajectory continues — but efficiency techniques (DeepSeek, distillation) are starting to push back.

Training cost structure

What makes up the $100 million

The estimates cited above cover almost exclusively compute cost — raw GPU calculation cost. The real structure is more complex:

Direct consequence: when Sam Altman cites “$100M for GPT-4”, he means compute cost. The total including teams, data, and infrastructure would be substantially higher — but those numbers aren’t public.

GPU tariffs in practice (2022–2026)

GPU cloud rates have been highly volatile:

• A100 80GB (2022-2024): $1.07–$3.43/GPU-hour depending on provider [Hyperstack, Spheron, TensorDock].
• H100 80GB SXM (2023-2025): $2.40–$3.00/GPU-hour on-demand; peak ~$8/GPU-hour during the 2023 shortage [Jarvis Labs, GMI Cloud, Thunder Compute].
• H100 80GB (2026): $1.38–$2.99/GPU-hour following supply improvement [Thunder Compute, March 2026].

The decline in GPU rates in 2025-2026, driven by supply expansion, mechanically makes future training runs cheaper — all else equal.

Pre-training vs fine-tuning: where the money goes

Training economics are highly concentrated:

• Pre-training: 95%+ of total compute. This is the base run on a massive corpus.
• Supervised fine-tuning + RLHF: a few percent of pre-training compute. Alignment and instruction-following add 5–20% depending on the model.
• Inference fine-tuning (LoRA, adapters): negligible, less than 1% of pre-training.

In plain terms: when people talk about GPT-4’s training cost, they mean 95% pre-training. Fine-tuning, alignment, specialization — all relatively cheap compared to the initial run.

Inference cost structure

Input vs output tokens: why prices are asymmetric

API pricing reflects the real cost structure of inference:

• Input tokens (prompt): cheaper. The model reads the provided context via the KV cache (a mechanism that stores attention computations for already-processed tokens). No sequential generation.
• Output tokens (completion): 3–6× more expensive. Generation is sequential — each token produced depends on the previous one. It consumes more GPU memory and time.

This asymmetry is visible in 2024 pricing:

Sources: OpenAI pricing page; Anthropic pricing page; SemiAnalysis DeepSeek Debates; IntuitionLabs 2025 comparison.

The KV cache: a silent economic lever

The KV cache (Key-Value cache) is the primary cost-reduction tool for long conversations. In Transformer architectures, each generated token requires computing attention over all previous tokens. The KV cache stores these intermediate computations and avoids recomputing them.

Providers monetize it differently. Anthropic, for example, offers a cache read rate 10× lower than new content: $0.30/M tokens on cache reads vs. $3/M tokens for fresh content (Claude Sonnet 2024). For applications with long, repetitive conversations — document assistants, enterprise chatbots — the savings can be substantial.

The 2024 flip: inference surpasses training

OpenAI in 2024: $2.3 billion in inference compute

OpenAI’s 2024 financial data illustrates the reversal concretely:

• 2024 revenue: approximately $6B (ARR June 2024: $3.4B; rapid growth).
• 2024 losses: ~$5B net despite those revenues [futuresearch.ai].
• 2024 inference spend: estimated at $2.3B in inference compute alone [TechCrunch / futuresearch.ai, 2024 — single source].

To put that in perspective: $2.3B in inference in 2024 represents approximately 15× the estimated training cost of GPT-4.5 [TechCrunch / futuresearch.ai crosscheck]. In a single year of operation, OpenAI spends in inference the equivalent of fifteen frontier model training cycles.

Anthropic: from -94% to +40% gross margin in two years

Anthropic’s data (Sacra.com; ai2.work) reveals a particularly striking trajectory:

• 2024 gross margin: -94%. Inference costs exceeded revenue generated.
• ARR December 2024: ~$1B.
• Projected gross margin 2025: ~40% [Sacra.com].
• ARR March 2026: ~$30B (growth partly driven by Claude Code at $2.5B ARR in February 2026).
• Projected cash-flow break-even: 2028 [futuresearch.ai].

A -94% gross margin in 2024 means Anthropic lost money on every dollar of revenue, purely from inference cost — before counting R&D, marketing, or headcount.

In plain terms: even at $15 per million tokens for Claude Opus, actual compute costs exceeded revenue. The explosion in usage (95% of Anthropic users are non-paying) [ai2.work] and cheap subscription models (ChatGPT Plus at $20/month) create a structural gap between listed price and real cost.

Why margins remain under pressure

Four mechanisms explain the margin squeeze:

1. “Unlimited” subscriptions underpriced: offering “unlimited” access at $20/month generates losses on heavy users whose consumption can exceed hundreds of dollars in actual compute.
2. Very high freemium ratio: 95% of Anthropic users don’t pay [ai2.work]. The paying base subsidizes a massive infrastructure.
3. GPU CAPEX escalating: cloud providers charge AI labs OPEX-style, passing on GPU rate spikes (H100 peaked at $8/hour during the 2023 shortage).
4. Reasoning models = costs ×10 to ×1,000: each deployment of a more “intelligent” model mechanically increases the cost of service per request.

Hyperscaler CAPEX: the structural escalation

Inference economics cannot be read without the broader infrastructure spending context:

Sources: IEEE ComSoc Technology Blog (Dec. 2025); Futurum Group (Apr. 2026); DCPulse; CNBC (Feb. 2026). 2026 figures are forward guidance declared by companies, not realized results.

Critical note: this CAPEX covers all cloud infrastructure, not AI alone. The AI/GPU share is estimated at 40–60% of the total in 2025-2026, up from 15–20% in 2022. The energy/cooling/real estate breakdown is covered in the [datacenters-energie] article.

In plain terms: in 2025, the four hyperscalers invested ~$350B in cloud infrastructure — a growing share in GPUs and inference systems. That’s 40 times the training budget of all major public models in the same year.

Reasoning models: the cost amplifier

The principle: spend at inference what you save on training

Reasoning models (o1, o3, R1) introduce a new economic paradigm. Rather than training a larger model to get better performance, they allocate additional computational resources at query time — via an internal “chain of thought” before responding.

Key terms

• Test-time compute (TTC): resources allocated at inference to improve response quality, as opposed to training compute. Also called “inference-time compute”.
• Chain of thought: an intermediate reasoning sequence generated by the model before its final answer. These “thinking” tokens consume compute but may not all be billed or visible.
• Reasoning model: a model specializing in TTC allocation — o1, o3 (OpenAI), R1 (DeepSeek). Distinct from standard “chat” models.

The economic consequence is direct: for the same complex task, a reasoning model consumes far more compute than a standard chat model. That surplus translates into price.

The orders of magnitude: from ×6 to ×1,000

The pricing table tells part of the story:

Sources: OpenAI pricing page (2024-2025); SemiAnalysis DeepSeek Debates.

Pricing captures only part of the real differential. Analysis of actual compute used reveals even more dramatic gaps:

• o1 vs GPT-4o-128k: factor ~15× inference compute for the same task [Poe.com analysis, cited in vellum.ai].
• o3-high vs o3-low: factor 170× in computational resources [TechCrunch, Maxwell Zeff, Dec. 23, 2024].
• o3 in ARC-AGI benchmark mode: “more than $10,000 to run the full benchmark in high mode” — too expensive to enter the ARC Prize [TechCrunch, 2024].
• o3-high vs o1-low: factor >1,000× [TechCrunch, 2024].

In plain terms: between o3’s economy mode and high-performance mode, OpenAI consumes 170 times more compute resources. This isn’t rounding error — it’s an economic chasm between two levels of the same model.

DeepSeek R1: disruption through efficiency

DeepSeek R1 (January 2025) delivers o1-equivalent performance at 20–50× lower cost. The key: architectural innovations on the inference side.

Two techniques are particularly economically significant:

1. MLA (Multi-head Latent Attention): KV cache compression via latent projections. Significantly reduces GPU memory required per request, lowering infrastructure cost [DeepSeek inference cost explained, IntuitionLabs].
2. Mixture of Experts (MoE): only a fraction of parameters activates per token. For DeepSeek V3, ~37B active parameters per token out of a total of 671B. Compute cost per token scales with active parameters, not the full model size.

The R1 announcement triggered a 17% single-day drop in Nvidia’s stock (approximately $600B in market cap erased) [HPCwire, Jan. 2025; SemiAnalysis DeepSeek Debates]. Market signal: if equivalent performance is achievable with far fewer GPUs, infrastructure demand may be overestimated.

The inference price collapse: -1,000× in three years

The a16z thesis: LLMflation

a16z documented in 2024 a cost reduction of 1,000× in three years for GPT-3-equivalent performance:

• November 2021: GPT-3 → $60 per million tokens.
• 2024: Llama 3.2 3B (MMLU ≥ 42 performance) → $0.06 per million tokens.

Reduction: 1,000 times in three years for the same benchmark performance level [a16z LLMflation, 2024].

Epoch AI: -50× per year median, -200× post-2024

Epoch AI (2025) analyzes inference price evolution across six benchmarks (MMLU, GPQA Diamond, MATH-500, HumanEval, Chatbot Arena):

• Median decline: 50× per year over 2022-2025 data.
• Post-January 2024 data only: 200× per year — the decline has accelerated [Epoch AI, data-insights, 2025].

For reference: “The price to achieve GPT-4’s performance on PhD-level science questions fell by 40× per year.”

Six structural drivers

Six mechanisms fuel this deflation:

1. Hardware improvement: from A100 to H100 to B200, each generation improves the FLOP/watt and FLOP/dollar ratio.
2. Quantization: reducing numerical precision (FP16 → FP8 → INT4) improves efficiency by a factor of ~4.
3. Software optimization: FlashAttention, PagedAttention, speculative decoding reduce memory bottlenecks without changing the model.
4. Smaller models outperforming older large ones: a 2024 Llama 3.2 3B beats 2020 GPT-3 at a fraction of the cost.
5. Instruction tuning and RLHF: better-aligned models need fewer tokens per task.
6. Open-source competition: Llama, Mistral, DeepSeek compress margins for closed providers.

The Jevons paradox

Despite the token cost collapse, total inference spending keeps growing. The 19th-century economist William Stanley Jevons observed that energy efficiency gains increased total consumption rather than reducing it — cheaper resources drive more usage.

The same dynamic is playing out in AI: cheaper tokens generate more applications, more users, more use cases, raising total volume. Industry projections [NON VÉRIFIÉ] estimate the AI inference market at $106B in 2025 and $255B in 2030 (19.2% CAGR).

Gartner projects a >90% drop in inference cost for one-trillion-parameter models by 2030 versus 2025 [Gartner, March 2026]. If the Jevons effect holds, that price drop will expand the market rather than contract it.

In plain terms: tokens cost a thousand times less than in 2021. Yet AI labs are spending more on inference than ever. Demand follows supply — volume explodes every time prices fall.

The 2025 pricing war

The DeepSeek shockwave in January 2025 triggered a competitive pricing race across the industry. Within months, per-token prices fell by more than 90% [SemiAnalysis DeepSeek Debates; HPCwire]. OpenAI, Anthropic, Google, and cloud providers all passed on or anticipated the pressure in their API pricing.

This creates a paradox for labs: cutting prices to stay competitive erodes already fragile margins, but not cutting prices pushes developers toward cheaper alternatives. Inference pricing has become a near-negative-sum game short-term, justified by the hope that volume compensates.

Economic strategy by use case

In plain terms: model selection and inference mode matter as much as the provider. Using o3-high where o3-low suffices multiplies your bill by 170.

Key takeaways for 2026

Generative AI economics are in a phase of structural tension. Training is expensive — up to $100M for a frontier model — but inference now dominates budgets over a model’s lifetime. OpenAI spent $2.3B on inference compute in 2024, approximately fifteen times the training cost of GPT-4.5. Anthropic posted a -94% gross margin the same year.

Simultaneously, the cost per token has collapsed — a factor of 1,000 in three years per a16z, -200× per year per Epoch AI since 2024. Reasoning models like o3 complicate the equation: they enable unprecedented performance but consume up to 1,000× more resources per request than their standard equivalents.

Pricing deflation (LLMflation) and volume growth (Jevons effect) are currently offsetting each other. The AI lab market remains unprofitable — losses run into the billions, justified by the prospect of an inflection point that projections place around 2028.