# Training compute

> Source: https://aiwiki.ai/wiki/training_compute
> Updated: 2026-10-15
> Categories: AI Compute, Machine Learning, Training & Optimization
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Training compute." aiwiki.ai, 15 Oct 2026. https://aiwiki.ai/wiki/training_compute
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Training compute** is the total number of floating-point operations ([FLOP](https://aiwiki.ai/wiki/flops)) performed to train a machine learning model, normally counted over the forward and backward passes of the final training run. It is a central variable in [scaling laws](https://aiwiki.ai/wiki/scaling_laws), which relate a model's loss to the compute, parameters and data used to train it, and it is the measure that the [EU AI Act](https://aiwiki.ai/wiki/eu_ai_act) and several US state laws use to decide which models face extra safety obligations.[1][2][3] By the estimates of the research group [Epoch AI](https://aiwiki.ai/wiki/epoch_ai), the compute used to train notable models rose from about 4.7 × 10^17 FLOP for AlexNet in 2012 to about 9.4 × 10^26 FLOP for [OpenAI](https://aiwiki.ai/wiki/openai)'s GPT-6 Astra in September 2026, roughly a two-billion-fold increase.[4] Epoch's February 2026 update put the growth in compute for frontier language models at about five times per year since 2020.[5]

## What does training compute measure?

Training compute is an amount of work rather than a speed. Chips are rated in FLOP per second (FLOP/s), and a run's total is the rate actually achieved, multiplied by the number of chips and the time they run. Early papers used the petaflop/s-day, the work of a machine running at 10^15 operations per second for a day, which equals 8.64 × 10^19 FLOP.[1][6] Epoch AI counts a multiply-add as two FLOP and makes no adjustment for numerical precision. The European Commission's guidelines on general-purpose AI models also tell providers to count all operations equally, so an operation in [FP8](https://aiwiki.ai/wiki/fp8) counts the same as one in a wider number format.[7][2]

Published figures usually describe a single run. Epoch's database records the compute of the run that produced a model, including any pretrained components, and lists fine-tuning compute in a separate field.[7] Developers follow a similar convention: Google's PaLM paper said its figures did not account for computation arising from development, evictions or other inefficiencies.[8] Experiments, ablations and abandoned runs therefore sit outside the headline number, even though Epoch concluded that such work probably took most of OpenAI's research compute in 2024, as described in the section on costs below.[9]

## How is training compute estimated?

Two methods dominate. One counts the arithmetic implied by a model's architecture and training data; the other multiplies the hardware used by the time it ran and the share of its peak speed it achieved. OpenAI used both in its 2018 study of compute trends, Epoch AI uses both in its database, and the European Commission's July 2025 guidelines accept either as long as the estimate is accurate to within 30%.[6][7][2]

### The 6ND approximation

For transformer language models, the usual shortcut comes from the January 2020 scaling-laws paper by Jared Kaplan and colleagues at OpenAI. A forward pass costs roughly 2N FLOP per token for a model with N non-embedding parameters, because each parameter takes part in one multiply and one add. The backward pass costs about twice as much, so training takes about 6N FLOP per token, and C ≈ 6ND for D training tokens.[1] The formula leaves out attention operations whose cost grows with context length. Kaplan's group treated them as a small share of the total when a model's width is large relative to its context window.[1]

Developers have used the same arithmetic to report their own runs. The [GPT-3](https://aiwiki.ai/wiki/gpt-3) paper multiplied 6 FLOP per parameter per token by 174.6 billion parameters and 300 billion tokens to get 3.14 × 10^23 FLOP, or 3,640 petaflop/s-days.[10] DeepMind's [Chinchilla](https://aiwiki.ai/wiki/chinchilla) paper compared 6ND with a fuller count that included attention and found ratios of 0.99 to 1.10 for models from 73 million to 6.8 billion parameters.[11] Meta's reported 3.8 × 10^25 FLOP for the 405-billion-parameter [Llama 3](https://aiwiki.ai/wiki/llama_3) model, trained on 15.6 trillion tokens, matches the formula.[12]

Mixture-of-experts models are counted by the parameters active for each token, not by their total size. [DeepSeek-V3](https://aiwiki.ai/wiki/deepseek_v3) has 671 billion parameters but activates 37 billion per token, and Epoch's figure of 6 × 37 billion × 14.8 trillion tokens gives about 3.3 × 10^24 FLOP.[13][4]

### The hardware method

The hardware-based estimate multiplies the number of chips by the training time, the peak FLOP/s of each chip and the average utilization. The Commission writes it as C = N · L · H · U, and Epoch uses the same product when it cannot count a model's operations directly.[2][7] OpenAI's 2018 analysis usually assumed 33% utilization when it had only the number of GPUs and the training time.[6]

Utilization is the hardest term to pin down. Google's PaLM paper proposed model FLOPs utilization (MFU), the observed throughput as a share of the hardware's theoretical maximum, and reported 46.2% MFU for PaLM 540B on 6,144 [TPU](https://aiwiki.ai/wiki/tpu) v4 chips.[8] Meta reported 38-43% MFU in [bfloat16](https://aiwiki.ai/wiki/bfloat16) for its 405-billion-parameter model, trained on up to 16,384 [H100](https://aiwiki.ai/wiki/nvidia_h100) GPUs.[12][4] Epoch's documentation treats 30-50% as typical for large distributed runs and borrows values from comparable models when a developer reports none.[7]

### Epoch AI's estimates and confidence levels

Some frontier developers no longer publish these inputs. OpenAI's [GPT-4](https://aiwiki.ai/wiki/gpt-4) technical report cited "both the competitive landscape and the safety implications of large-scale models like GPT-4" as its reason for giving no further details about the architecture, hardware, training compute or training method.[14] Outside estimates fill the gap. Epoch AI's database, which tracked more than 3,600 models as of October 2026, uses a reported figure where one exists, prefers operation counting when the architecture and data are known, falls back on hardware details, and for the least documented models infers compute from benchmark scores.[4][7]

Each estimate carries a confidence label that stands for a 90% interval. "Confident" means within a factor of 3, "Likely" within a factor of 10, and "Speculative" within a factor of about 31, or 1.5 orders of magnitude.[7] Epoch's figure for GPT-4, 2.1 × 10^25 FLOP, is rated Likely, with a stated interval of 8.2 × 10^24 to 4.4 × 10^25 FLOP.[4]

## How fast has training compute grown?

### From "AI and Compute" to three eras

The first widely cited trend came from Dario Amodei and Danny Hernandez of OpenAI. Their post "AI and Compute", published on May 16, 2018, found that the compute used in the largest training runs had doubled every 3.4 months since 2012, an increase of more than 300,000 times between AlexNet and AlphaGo Zero, where a two-year doubling period would have produced about a sevenfold increase. An addendum covering 1959 to 2012 found a doubling time of about two years for that earlier period.[6]

In February 2022, Jaime Sevilla and colleagues at Epoch revisited the question with more than 120 milestone systems, about ten times as much data. "Compute Trends Across Three Eras of Machine Learning" found a doubling time of about 20 months before deep learning and about 6 months in the deep learning era that began between 2010 and 2012. It also identified a separate large-scale trend that broke off between 2015 and 2016, with models using 10 to 100 times more compute and doubling about every 10 months.[15]

### The 4-5x per year trend

Epoch's May 2024 update, by Jaime Sevilla and Edu Roldán, found that the training compute of notable models grew about 4.1 times per year from 2010 to May 2024 (90% confidence interval 3.7-4.6), while the running top 10 frontier models grew about 5.3 times per year. Frontier language models grew about five times per year after a slowdown around mid-2020.[16] Epoch's trends page, last updated on February 5, 2026, gives five times per year since 2020 for frontier language models (90% interval 4-6), a doubling about every 5.2 months.[5]

Epoch counted more than 30 publicly announced models from 12 developers above 10^25 FLOP by June 2025, with about two new ones a month during 2024.[17] The first model in its data estimated above 10^26 FLOP was [xAI](https://aiwiki.ai/wiki/xai)'s Grok 3, released in February 2025. As of October 2026 the database gives central estimates above 10^26 FLOP for four models: Grok 3, GPT-4.5, Grok 4 and GPT-6 Astra. It has no estimate at all for many flagship models from [Anthropic](https://aiwiki.ai/wiki/anthropic) and Google, so counts of models above a given threshold are lower bounds.[4]

### Training compute of notable models

The table gives reported figures where developers published them and Epoch AI estimates otherwise, as listed in Epoch's database in October 2026. Estimates for undisclosed models can change when new information appears.[4]

| Model | Developer | Released | Training compute (FLOP) | log10 FLOP | Basis |
|---|---|---|---|---|---|
| AlexNet | University of Toronto | 2012-09 | 4.7 × 10^17 | 17.7 | Epoch estimate, Confident[4] |
| GPT-2 (1.5B) | OpenAI | 2019-02 | 1.9 × 10^21 | 21.3 | Epoch estimate, Speculative[4] |
| GPT-3 175B | OpenAI | 2020-05 | 3.14 × 10^23 | 23.5 | Reported by OpenAI[10] |
| Chinchilla 70B | DeepMind | 2022-03 | 5.76 × 10^23 | 23.8 | Reported by DeepMind[11] |
| PaLM 540B | Google | 2022-04 | 2.53 × 10^24 | 24.4 | Reported by Google[8] |
| GPT-4 | OpenAI | 2023-03 | 2.1 × 10^25 | 25.3 | Epoch estimate, Likely; not disclosed by OpenAI[4] |
| Gemini 1.0 Ultra | Google DeepMind | 2023-12 | 5 × 10^25 | 25.7 | Epoch estimate, Speculative[4] |
| Llama 3.1 405B | Meta | 2024-07 | 3.8 × 10^25 | 25.6 | Reported by Meta[12] |
| Grok 2 | xAI | 2024-08 | 3.0 × 10^25 | 25.5 | Epoch estimate, Confident[4] |
| DeepSeek-V3 | DeepSeek | 2024-12 | 3.3 × 10^24 | 24.5 | Epoch estimate from reported architecture and 2.788 million H800 GPU-hours, Confident[13][4] |
| Grok 3 | xAI | 2025-02 | 3.5 × 10^26 | 26.5 | Epoch estimate, Likely[4] |
| GPT-4.5 | OpenAI | 2025-02 | 3.8 × 10^26 | 26.6 | Epoch estimate, Likely[4] |
| Grok 4 | xAI | 2025-07 | 5 × 10^26 | 26.7 | Epoch estimate, Speculative[4] |
| GPT-5 | OpenAI | 2025-08 | 6.6 × 10^25 | 25.8 | Epoch estimate, Speculative[4] |
| DeepSeek-V4-Pro | DeepSeek | 2026-04 | 9.7 × 10^24 | 25.0 | Epoch estimate, Likely[4] |
| GPT-6 Astra | OpenAI | 2026-09 | 9.4 × 10^26 | 27.0 | Epoch estimate, Likely[4] |

## Compute-optimal training and over-training

A training budget can be spent on a bigger model or on more data, and the preferred split has changed. Kaplan's 2020 fits implied that, as compute grows, the optimal model size should grow roughly as C^0.73 and the number of training tokens only as C^0.27, which favored very large models trained on relatively little data.[11] GPT-3, with 175 billion parameters, was trained on 300 billion tokens.[10]

In March 2022, Jordan Hoffmann and colleagues at DeepMind trained more than 400 models and concluded that "for every doubling of model size the number of training tokens should also be doubled". Their projections work out to about 20 tokens per parameter, for example 20.2 billion tokens for a 1-billion-parameter model and 1.5 trillion tokens for 67 billion parameters. Chinchilla, a 70-billion-parameter model trained on 1.4 trillion tokens, used the same 5.76 × 10^23 FLOP as the 280-billion-parameter Gopher and outperformed it on a wide range of evaluations.[11]

Compute-optimal in this sense means the lowest training loss for a given training budget, without regard to what the model costs to run afterwards. Developers who expect heavy use now train smaller models for far longer. Meta's April 2024 Llama 3 announcement said the Chinchilla-optimal amount of training compute for an 8-billion-parameter model corresponded to about 200 billion tokens, yet its 8B and 70B models kept improving log-linearly up to 15 trillion tokens, about 75 times that amount in the 8B case.[18] Meta's technical report said it trained the smaller models "much longer than is compute-optimal" so they would perform better at a given [inference](https://aiwiki.ai/wiki/inference) budget, while the 405B flagship was close to the size its scaling experiments predicted for the budget, 402 billion parameters on 16.55 trillion tokens.[12] As a result, a training compute figure no longer indicates model size on its own: the same FLOP can produce a large model trained briefly or a small one trained on many more tokens.

## How much does a frontier training run cost?

Epoch AI estimated in June 2024 that the amortized hardware and energy cost of the final training run for the most compute-intensive models had grown about 2.4 times per year since 2016, and projected that the largest runs would cost more than $1 billion by 2027. For models such as GPT-4 and Gemini, hardware made up 47-67% of development costs, research staff 29-49% and energy only 2-6%. Pricing the same runs at cloud rental rates gave estimates about twice as high, because many frontier models are trained on hardware their developers own.[19] Epoch's February 2026 trends update put the growth of frontier training costs at about 3.5 times per year since 2020 (90% interval 2.8-4.4).[5]

| Model | Year | Estimated cost of final training run | Basis |
|---|---|---|---|
| GPT-4 | 2023 | about $37 million | Epoch database; amortized hardware and energy, 2023 dollars[4] |
| Gemini 1.0 Ultra | 2023 | about $31 million | Epoch database; same method[4] |
| Llama 3.1 405B | 2024 | about $53 million | Epoch database; same method[4] |
| DeepSeek-V3 | 2024 | $5.576 million | DeepSeek; 2.788 million H800 GPU-hours at an assumed $2 per hour[13] |
| GPT-4.5 | 2025 | about $400 million (90% interval $170-890 million) | Epoch; 40,000-100,000 H100s for 90-165 days at $1.50-3 per H100-hour[9] |
| Grok 4 | 2025 | about $490 million | Epoch; about 246 million H100-hours, median of rental and depreciation methods[20] |

Company statements are rare and differ in scope. Asked at an MIT event in April 2023 whether training GPT-4 had cost $100 million, OpenAI chief executive Sam Altman replied "It's more than that", according to Wired.[21] Anthropic chief executive Dario Amodei wrote in January 2025 that Claude 3.5 Sonnet, a mid-sized model, "cost a few $10M's to train", without giving an exact figure.[22]

[DeepSeek](https://aiwiki.ai/wiki/deepseek)'s December 2024 report priced the training of DeepSeek-V3, 2.788 million H800 GPU-hours on a cluster of 2,048 GPUs, at $5.576 million, assuming a rental price of $2 per GPU-hour. It said the figure covered only the official training run and excluded "the costs associated with prior research and ablation experiments on architectures, algorithms, or data".[13] The research firm [SemiAnalysis](https://aiwiki.ai/wiki/semianalysis) estimated in January 2025 that DeepSeek had access to around 50,000 Hopper-generation GPUs and about $1.6 billion in server capital expenditure, and described the widely quoted $6 million as the GPU cost of the pretraining run alone, leaving out research and development and the total cost of owning the hardware.[23]

Final runs can be a small part of what a developer spends on compute. Using figures reported by The Information and The New York Times, Epoch estimated that OpenAI spent about $7 billion on compute in 2024, about $5 billion of it on research and development and about $2 billion on inference. Only about $470 million of the research spending, roughly 10%, went to the final training runs of models the company released, about $400 million of that for GPT-4.5, and Epoch concluded that most of the rest likely went to research, experimental runs and unreleased models. The underlying figures came from investor documents that partly projected the fourth quarter.[9]

## Training clusters and hardware scale

Epoch's April 2025 study of AI supercomputers found that the computational performance of the largest systems grew about 2.5 times per year, with chip counts rising 1.6 times, hardware costs 1.9 times and power needs 2 times per year. The largest, xAI's [Colossus](https://aiwiki.ai/wiki/xai_colossus) in Memphis, had about 200,000 AI chips, an estimated $7 billion of hardware and about 300 MW of power as of March 2025. Companies' share of AI supercomputer capacity rose from about 40% in 2019 to 80% in 2025, and the United States held about 75% of the computing power in the dataset.[24]

| Cluster or run | Operator | Date | Accelerators | Source |
|---|---|---|---|---|
| PaLM 540B training | Google | 2022 | 6,144 TPU v4 chips across two pods | [8] |
| Llama 3 training clusters | Meta | 2024 | two custom-built clusters of about 24,000 GPUs each; Llama 3.1 405B used up to 16,384 H100s | [18][12] |
| DeepSeek-V3 training | DeepSeek | 2024 | 2,048 H800 GPUs | [13] |
| Colossus | xAI | 2025 | about 200,000 GPUs; used for Grok 4's reinforcement learning | [24][25] |
| Project Rainier | Amazon Web Services for Anthropic | 2025 | nearly 500,000 Trainium2 chips | [26] |
| Stargate site in Texas | OpenAI | 2026 | more than 100,000 GPUs used to pretrain GPT-6 Astra | [27] |
| Colossus 2 | xAI | 2026 | about 1.1 million H100-equivalents (Epoch estimate) | [5] |
| Google TPU fleet | Google | 2026 | training spread across more than 1 million TPUs at several sites | [28] |

Each step up in cluster size has brought engineering problems of its own. Meta trained the Llama 3 models on two custom-built [GPU clusters](https://aiwiki.ai/wiki/gpu_cluster) and reported an effective training time of more than 95%, yet its technical report recorded 466 job interruptions during a 54-day period of pretraining the 405B model, 419 of them unexpected.[18][12] Amazon Web Services activated Project Rainier in October 2025 and said its [Trainium2](https://aiwiki.ai/wiki/aws_trainium2) chips gave Anthropic more than five times the compute it had used to train its previous models.[26]

OpenAI researcher Aidan Clark called the GPT-6 Astra run, which used more than 100,000 GPUs at the company's [Stargate](https://aiwiki.ai/wiki/stargate_project) site in Texas, OpenAI's largest training run.[27] Epoch's trends page lists xAI's [Colossus 2](https://aiwiki.ai/wiki/xai_colossus_2) in Memphis as the largest known [AI data center](https://aiwiki.ai/wiki/ai_datacenter), and Google said at its I/O conference in May 2026 that it could spread a single training run across TPUs at several sites.[5][28]

## Training versus inference compute

Until recently, Epoch AI wrote in 2025, most large language models were trained with about 100 times more pretraining compute than post-training compute.[29] Reasoning models changed that by spending large amounts of compute on reinforcement learning after pretraining and on long chains of reasoning when answering, known as [test-time compute](https://aiwiki.ai/wiki/test_time_compute). xAI said it used its 200,000-GPU Colossus cluster to run reinforcement learning for Grok 4 "at pretraining scale", and that the run used over an order of magnitude more compute than had been used previously.[25]

The shift showed up in Epoch's estimates. In September 2025 Epoch concluded that GPT-5 had probably been trained with less compute than GPT-4.5, unlike every earlier GPT model, because OpenAI focused on scaling post-training, which offered better returns at the margin. It gave a rough estimate that new reasoning techniques allowed pretraining compute to be cut about tenfold for the same performance, and predicted that GPT-6 would probably need more training compute than GPT-4.5.[29] Its later estimate for GPT-6 Astra, 9.4 × 10^26 FLOP, is the largest in its database.[4]

Spending on inference is of the same order as spending on training. Epoch's estimate for OpenAI in 2024 put about $2 billion into inference against about $5 billion for research, experiments and training.[9] Epoch's "AI in 2030" report, commissioned by Google DeepMind and published in September 2025, argued that training and inference compute had stayed within the same order of magnitude at large developers, and that growth in inference need not slow growth in training because reasoning models also benefit from more training compute.[30] Deloitte forecast in November 2025 that inference would make up two-thirds of AI compute by 2026.[31]

Inference demand has risen quickly. Google said in May 2026 that it was processing more than 3.2 quadrillion tokens a month across its products, up from about 480 trillion a year earlier and 9.7 trillion in May 2024. At the same event it introduced its eighth-generation TPUs as two chips, one specialized for training and one for inference.[28]

## How do regulators use training compute thresholds?

Training compute is the usual trigger in frontier AI rules, an approach discussed under [compute governance](https://aiwiki.ai/wiki/compute_governance). One reason is that it can be estimated before a model exists: the Commission's guidelines note that training is planned with an upfront allocation of compute, so a provider can know before training ends whether a model will cross the threshold, and they ask providers to estimate the figure before the large pretraining run begins.[2]

| Jurisdiction | Instrument | Compute threshold | Main effect | Status as of October 2026 |
|---|---|---|---|---|
| European Union | AI Act, Article 51(2) | more than 10^25 FLOP of cumulative training compute | Presumption that a general-purpose model has high-impact capabilities (systemic risk); notify the Commission within two weeks | Obligations apply from 2025-08-02; Commission enforcement powers from 2026-08-02[2] |
| European Union | Commission guidelines of 2025-07-18 | more than 10^23 FLOP | Indicative criterion that a model able to generate text, audio, images or video is a general-purpose AI model | Non-binding guidance[2] |
| United States | Executive Order 14110 | more than 10^26 operations (10^23 for models trained mainly on biological sequence data); clusters above 10^20 operations per second | Reporting to the federal government | Signed 2023-10-30; revoked 2025-01-20[32][33] |
| California | SB 53, Transparency in Frontier Artificial Intelligence Act | more than 10^26 operations, including later fine-tuning, reinforcement learning and material modifications | Transparency reports and 15-day incident reporting; published frontier AI frameworks for developers with revenue above $500 million | Signed 2025-09-29; in effect since 2026-01-01[3][34] |
| New York | RAISE Act, as amended | more than 10^26 operations, including fine-tuning and reinforcement learning | Safety protocols and 72-hour incident reporting; $500 million revenue tier | Amendments signed 2026-03-27; effective 2027-01-01[34] |
| Illinois | SB 315, Artificial Intelligence Safety Measures Act | more than 10^26 operations | Annual independent third-party audits for developers with revenue above $500 million | Signed 2026-07-06; first provisions effective 2027-01-01[35] |
| South Korea | AI Basic Act and Enforcement Decree | at least 10^26 FLOP of cumulative training compute | Life-cycle risk management and reporting to the science ministry for "high-performance" AI | In force since 2026-01-22; one-year grace period for fines[36] |

The Commission's guidelines spell out what "cumulative" compute covers. Providers should count all compute that contributes to a model's capabilities, including the generation of non-public synthetic training data (discarded samples included) and the training of weights that are merged into the model. They need not count publicly accessible synthetic data, evaluations and red-teaming, exploratory research and failed experiments, the training of parent models used for distillation or of reward models, or the recomputation of activations to save memory. Notifications report the figure to two significant figures.[2]

In the United States, [Executive Order 14110](https://aiwiki.ai/wiki/executive_order_14110) set the first federal reporting threshold at 10^26 operations in October 2023. It was revoked on January 20, 2025, and later state laws adopted the same number.[32][33] California's SB 53 also requires the state's Department of Technology to review its definitions every year, starting by January 1, 2027.[3] New York's amended RAISE Act dropped an earlier test of $100 million in compute costs, and Illinois became the first state to require annual independent third-party audits of large frontier developers' safety practices.[34][35]

The thresholds are rough proxies. Sara Hooker argued in July 2024 that "compute thresholds, as currently implemented, are shortsighted and likely to fail to mitigate risk", in part because the relationship between compute and risk is highly uncertain and rapidly changing.[37] They also rely on numbers that outsiders cannot check: Epoch's speculative estimate for GPT-5, about 6.6 × 10^25 FLOP with an interval reaching 2 × 10^26, straddles the 10^26 line used by the state laws.[4] Each definition counts compute used to train or modify a model, so the compute that a reasoning model spends at inference time does not count toward any of the thresholds.[3][2]

## What limits further growth?

Power is the most visible constraint, and training is one part of rising [AI energy consumption](https://aiwiki.ai/wiki/ai_energy_consumption). A white paper by EPRI and Epoch AI in August 2025 estimated that frontier training runs drew about 100-150 MW each and projected 1-2 GW per run by 2028 and more than 4 GW by 2030. The peak power of frontier runs had grown about 2.2 times per year from 2018 to 2025, and the paper noted that training spread across data centers 15-50 miles apart had already been demonstrated, which could ease local limits.[38] Epoch put the energy used to train Grok 4 at about 310 GWh.[20]

Data is another constraint. Epoch's 2024 estimate put the effective stock of public human-generated text at about 300 trillion tokens (90% interval 100-1,000 trillion) and projected that it would be fully used at some point between 2026 and 2032, or by 2027 if models are over-trained fivefold.[39] The "AI in 2030" report judged that there was enough public text to scale until at least 2027 and that synthetic and multimodal data made the bottleneck look surmountable.[30]

Epoch's August 2024 analysis "Can AI scaling continue through 2030?" concluded that training runs of about 2 × 10^29 FLOP would likely be feasible by 2030. It estimated that data center campuses of 1-5 GW were likely possible by then, that chip production could supply about 100 million H100-equivalents for training (range 20-400 million), and that latency would cap training runs only at 3 × 10^30 to 10^32 FLOP.[40] The September 2025 "AI in 2030" report projected that, on current trends, the largest models of 2030 would use about 10^29 FLOP, roughly 1,000 times the compute of the largest models of 2025, with training clusters costing more than $100 billion and drawing gigawatts of power.[30] Epoch's supercomputer study likewise projected that the leading system in June 2030 would have about 2 million chips, cost about $200 billion and need about 9 GW.[24]

The "AI in 2030" report described investment as a major uncertainty that interacts with all of the other bottlenecks.[30] Alphabet said in May 2026 that it expected [capital expenditure](https://aiwiki.ai/wiki/ai_capex) of about $180-190 billion that year, roughly six times its 2022 level.[28] Amodei guessed in January 2025 that algorithmic progress was improving efficiency by about 4 times per year, and wrote that "the gains in cost efficiency end up entirely devoted to training smarter models".[22]

## References

1. Jared Kaplan et al. "Scaling Laws for Neural Language Models." arXiv, 2020-01-23. https://arxiv.org/abs/2001.08361
2. "Guidelines on the scope of the obligations for providers of general-purpose AI models under the AI Act." European Commission, 2025-07-18. https://digital-strategy.ec.europa.eu/en/library/guidelines-scope-obligations-providers-general-purpose-ai-models-under-ai-act
3. "Senate Bill No. 53, Chapter 138: Transparency in Frontier Artificial Intelligence Act." California Legislature, 2025-09-29. https://leginfo.legislature.ca.gov/faces/billTextClient.xhtml?bill_id=202520260SB53
4. "Data on AI models" (database). Epoch AI, 2026-10-15. https://epoch.ai/data/ai-models
5. "Trends in AI." Epoch AI, 2026-02-05. https://epoch.ai/trends
6. Dario Amodei and Danny Hernandez. "AI and Compute." OpenAI, 2018-05-16. https://openai.com/index/ai-and-compute/
7. "AI models documentation: estimation." Epoch AI, accessed 2026-10-15. https://epoch.ai/data/ai-models-documentation/estimation
8. Aakanksha Chowdhery et al. "PaLM: Scaling Language Modeling with Pathways." arXiv, 2022-04-05. https://arxiv.org/abs/2204.02311
9. Josh You. "Most of OpenAI's 2024 compute went to experiments." Epoch AI, 2025-10-10. https://epoch.ai/data-insights/openai-compute-spend
10. Tom B. Brown et al. "Language Models are Few-Shot Learners." arXiv, 2020-05-28. https://arxiv.org/abs/2005.14165
11. Jordan Hoffmann et al. "Training Compute-Optimal Large Language Models." arXiv, 2022-03-29. https://arxiv.org/abs/2203.15556
12. Aaron Grattafiori et al. "The Llama 3 Herd of Models." arXiv, 2024-07-31. https://arxiv.org/abs/2407.21783
13. DeepSeek-AI. "DeepSeek-V3 Technical Report." arXiv, 2024-12-27. https://arxiv.org/abs/2412.19437
14. OpenAI. "GPT-4 Technical Report." arXiv, 2023-03-15. https://arxiv.org/abs/2303.08774
15. Jaime Sevilla et al. "Compute Trends Across Three Eras of Machine Learning." Epoch AI, 2022-02-16. https://epoch.ai/blog/compute-trends
16. Jaime Sevilla and Edu Roldán. "Training compute of frontier AI models grows by 4-5x per year." Epoch AI, 2024-05-28. https://epoch.ai/blog/training-compute-of-frontier-ai-models-grows-by-4-5x-per-year
17. "Over 30 AI models have been trained at the scale of GPT-4." Epoch AI, 2025-01-30 (updated 2025-06-06). https://epoch.ai/data-insights/models-over-1e25-flop
18. "Introducing Meta Llama 3: The most capable openly available LLM to date." Meta AI, 2024-04-18. https://ai.meta.com/blog/meta-llama-3/
19. Ben Cottier, Robi Rahman, Loredana Fattorini, Nestor Maslej and David Owen. "How much does it cost to train frontier AI models?" Epoch AI, 2024-06-03 (updated 2025-01-13). https://epoch.ai/blog/how-much-does-it-cost-to-train-frontier-ai-models
20. James Sanders, Luke Emberson and Yafah Edelman. "What did it take to train Grok 4?" Epoch AI, 2025-09-12. https://epoch.ai/data-insights/grok-4-training-resources
21. Will Knight. "OpenAI's CEO Says the Age of Giant AI Models Is Already Over." Wired, 2023-04-17. https://www.wired.com/story/openai-ceo-sam-altman-the-age-of-giant-ai-models-is-already-over/
22. Dario Amodei. "On DeepSeek and Export Controls." darioamodei.com, 2025-01-29. https://www.darioamodei.com/post/on-deepseek-and-export-controls
23. Dylan Patel et al. "DeepSeek Debates: Chinese Leadership On Cost, True Training Cost, Closed Model Margin Impacts." SemiAnalysis, 2025-01-31. https://newsletter.semianalysis.com/p/deepseek-debates
24. Konstantin F. Pilz, Robi Rahman, James Sanders and Lennart Heim. "Trends in AI Supercomputers." Epoch AI, 2025-04-23. https://epoch.ai/blog/trends-in-ai-supercomputers
25. "Grok 4." xAI, 2025-07-09. https://x.ai/news/grok-4
26. "AWS activates Project Rainier: One of the world's largest AI compute clusters comes online." Amazon, 2025-10-29. https://www.aboutamazon.com/news/aws/aws-project-rainier-ai-trainium-chips-compute-cluster
27. "GPT-6 Astra is the first model making OpenAI willing to declare the 'AGI era'." The Decoder, 2026-09-03. https://the-decoder.com/gpt-6-astra-is-the-first-model-making-openai-willing-to-declare-the-agi-era/
28. Sundar Pichai. "Google I/O 2026: Sundar Pichai's opening keynote." Google, 2026-05-19. https://blog.google/innovation-and-ai/sundar-pichai-io-2026/
29. Yafah Edelman, Jean-Stanislas Denain, Jaime Sevilla and Anson Ho. "Why GPT-5 used less training compute than GPT-4.5 (but GPT-6 probably won't)." Epoch AI, 2025-09-26. https://epoch.ai/gradient-updates/why-gpt5-used-less-training-compute-than-gpt45-but-gpt6-probably-wont
30. Epoch AI. "AI in 2030: Extrapolating current trends." Report commissioned by Google DeepMind, 2025-09-16. https://epoch.ai/files/AI_2030.pdf
31. "Deloitte 2026 Technology, Media & Telecommunications Predictions: Narrowing the gap between the promise of AI and its reality." Deloitte, 2025-11-18. https://www.deloitte.com/global/en/about/press-room/2026-tmt-predictions.html
32. "Executive Order on the Safe, Secure, and Trustworthy Development and Use of Artificial Intelligence." The White House, 2023-10-30. https://bidenwhitehouse.archives.gov/briefing-room/presidential-actions/2023/10/30/executive-order-on-the-safe-secure-and-trustworthy-development-and-use-of-artificial-intelligence/
33. "Initial Rescissions of Harmful Executive Orders and Actions: Communication from the President of the United States" (House Document 119-28). US Government Publishing Office, 2025-03-17. https://www.govinfo.gov/content/pkg/CDOC-119hdoc28/html/CDOC-119hdoc28.htm
34. "New York's Frontier AI Law Gets a California Makeover, With Some Key Differences." Cooley, 2026-04-01. https://www.cooley.com/news/insight/2026/2026-03-31-new-yorks-frontier-ai-law-gets-a-california-makeover-with-some-key-differences
35. "Illinois Governor Signs Frontier AI Model Law." Hunton Andrews Kurth, 2026-07-17. https://www.hunton.com/privacy-and-cybersecurity-law-blog/illinois-governor-signs-frontier-ai-model-law
36. "South Korea's AI Basic Act: Overview and Key Takeaways." Cooley, 2026-01-27. https://www.cooley.com/news/insight/2026/2026-01-27-south-koreas-ai-basic-act-overview-and-key-takeaways
37. Sara Hooker. "On the Limitations of Compute Thresholds as a Governance Strategy." arXiv, 2024-07-08. https://arxiv.org/abs/2407.05694
38. EPRI and Epoch AI. "Scaling Intelligence: The Exponential Growth of AI's Power Needs." EPRI white paper, 2025-08-11. https://restservice.epri.com/publicattachment/94532
39. Pablo Villalobos et al. "Will we run out of data? Limits of LLM scaling based on human-generated data." Epoch AI, 2024-06-06. https://epoch.ai/blog/will-we-run-out-of-data-limits-of-llm-scaling-based-on-human-generated-data
40. Jaime Sevilla et al. "Can AI scaling continue through 2030?" Epoch AI, 2024-08-20. https://epoch.ai/blog/can-ai-scaling-continue-through-2030