Inference Economics

Inference economics is the study of the resources, costs and revenue associated with running a trained AI model. It connects the price of compute with the work a serving system delivers. A rental rate, an API token price and a cost per completed task describe different parts of that system. Comparing deployment options requires a defined workload and service requirements.^[1][2]

Units and scope

An hourly GPU price describes access to hardware over time. It does not state how many requests the system can complete. Throughput depends on the model, serving software, precision, input and output lengths, concurrency and hardware configuration. A benchmark must also state its latency requirement: a configuration that produces more total tokens may make each user wait longer.^[1]

Published benchmark results have their own scope. MLPerf Inference, for example, defines workloads, scenarios, model requirements and reproducibility rules. A result under one benchmark's rules cannot establish performance on every application.^[2]

From billed hours to delivered tokens

Let H be the total hosting cost per billed hour, q the aggregate output-token throughput during active serving, and u the productive serving fraction of billed time. For a fixed workload, a simplified hosting cost per million output tokens is:

H * 1,000,000 / (3,600 * q * u)

This is an accounting identity for the stated inputs, not a performance prediction. Input processing must already be included in the throughput measurement. If measured throughput uses the entire paid wall-clock interval, including idle time and restarts, a separate utilization penalty would count those losses twice. The calculation also needs a separate treatment for any costs excluded from H.^[1][3]

For a hypothetical system costing $6 per billed hour and producing 1,000 output tokens per second during active serving, the formula gives about $1.67 per million output tokens at full productive utilization. At 50% productive utilization it gives about $3.33. These are arithmetic examples with assumed inputs, not rental quotes or measured model results.

API prices and self-hosting

An API bill can include different rates for uncached input, cached input and output. The amount of each category matters as much as its unit price. Reasoning models can consume additional output or reasoning tokens, so two models charging the same price per token can have different costs on the same task. Artificial Analysis calculates its cost per task using token consumption across its evaluation workload; this is a benchmark-specific measure.^[4]

A self-hosting comparison needs the cost of the whole deployment. GPU rent alone may omit CPU, memory, storage, network traffic, software and operations. Owned hardware requires a cost-allocation method rather than treating its purchase price as the cost of one run. GPU depreciation and cash expenditure are separate measures.^[1]

The options should be evaluated against the same acceptance and latency requirements. A cheaper output that needs more retries or human correction can have a higher AI task cost. Conversely, a larger or more expensive model may be unnecessary if a smaller system meets the task's requirements.

Utilization, interruptions and commitments

Productive utilization is not the same as a GPU's hardware-utilization counter. A reserved server can be billed while no requests arrive. An interruptible server can lose work or incur restart time. Azure's Spot VM documentation explicitly distinguishes its interruptible capacity from a service with availability guarantees.^[5]

Commitment terms also affect comparisons. In its November 2, 2026 registration-statement amendment, Anthropic reported that short-term compute cost materially more than comparable compute procured under multi-year commitments.^[6] A lower committed hourly rate should therefore be assessed alongside minimum duration, prepaid credits, cancellation terms and unused capacity. Prepaid credit applied to a future invoice is not an additional charge to add to that same invoice.

Revenue, contribution and profit

A serving provider's revenue depends on the work it sells and the prices customers actually pay. A contribution calculation subtracts the stated serving costs from that revenue. It does not establish net profit if research, administration, financing or other expenses remain outside the calculation.

The break-even productive utilization for a fixed-capacity scenario is the utilization at which included revenue equals included costs. A scenario that requires utilization above 100% cannot break even under those assumptions. Missing throughput or selling-price evidence should remain unknown rather than being replaced with a favorable estimate.

Reading price indices

A GPU price index and a token-price index can describe different customer populations and contract types. Ornn's OCPI methodology measures on-demand transactions and excludes bespoke long-term contracts. A comparison with another series must retain those scope differences.^[7]

Lower unit prices also do not imply lower total spending: expenditure depends on both price and purchased volume. Platform token counts are another distinct measurement. OpenRouter's rankings count tokens processed through its own API, exclude private requests and do not constitute a census of worldwide AI spending.^[8]

Continue reading

• MModelA model in machine learning is a representation used to map inputs to outputs, assign probabilities, describe relationships, or choose actions.Mentioned 7 times above
• AAI computeAI compute is the processing capacity used to train and run artificial intelligence models, together with the chips, memory, networks, data centers, electricity and capital that supply it.Mentioned 3 times above
• GGPU DepreciationGPU depreciation concerns the allocation of the cost of graphics processing units over their useful lives and the loss of their economic value.Links back to this article
• AAI Task CostAI task cost measures the spending associated with executing a defined task using an AI system. It can be expressed per attempt, completed workflow or accepted output.Links back to this article
• CCompute ScarcityCompute scarcity is a shortage of usable computing capacity relative to demand under specified technical, commercial and delivery conditions.Also in AI Compute
• DData CenterA data center is a purpose-built facility, or a dedicated part of a facility, that houses and interconnects information technology and telecommunications equipment together with the power…Also in AI Compute

More to read

• Google Cloud
• AI Infrastructure
• NVIDIA H100
• Amazon Web Services
• AI Chip
• AI Accelerator

Learning path: Understand AI compute economics

Compare a fixed workload with explicit service terms, utilization, latency and success criteria.

Continue with: Define a successful task →Earlier on this path: Read the service and contract terms

Topics:AI ComputeAI Infrastructure

References

1. ^1 ^2 ^3 ^4Vinh Nguyen and Sergio Perez. "LLM Inference Benchmarking: How Much Does Your LLM Inference Cost?" NVIDIA Technical Blog, June 18, 2025.
2. ^1 ^2MLCommons. "MLPerf Inference Rules." Benchmark policies, accessed November 3, 2026.
3. ^vLLM Project. "Benchmark Suites." Documentation, accessed November 3, 2026.
4. ^Artificial Analysis. "Artificial Analysis Benchmarking Methodology." Definitions of price, tokens and cost per task, accessed November 3, 2026.
5. ^Microsoft. "Azure Spot Virtual Machines." Microsoft Learn, accessed November 3, 2026.
6. ^Anthropic, PBC. Form S-1/A. Filed November 2, 2026, management discussion of compute commitments and quarterly results.
7. ^Ornn Data. "Ornn Compute Price Indices (OCPI): Methodology." Universe creation, accessed November 3, 2026.
8. ^OpenRouter. "AI Model Rankings." Measurement methodology, accessed November 3, 2026.