Scaling Laws

Neural scaling laws are empirical relationships that describe how a measured outcome changes as a neural network, its training data, or the computation used to train it becomes larger. In a fixed experimental setting, quantities such as held-out cross-entropy loss often improve approximately as a power of parameter count, data size, or training compute. The relationship is called a scaling law because it can be fitted on smaller runs and used to estimate larger ones.

These relationships are regularities of particular experiments, not universal laws of nature. Their coefficients depend on what is counted as a parameter or a token, the data distribution, tokenizer, architecture, optimizer, learning-rate schedule, numerical precision, training horizon, and metric. A curve that is accurate inside one measured range can fail after a change in any of those conditions or after the system enters a different scaling regime.^[1][2]

Scaling laws are used in machine learning to plan experiments and allocate a compute budget among model size, data, and training duration. They have also been studied in vision, multimodal modeling, transfer learning, routed models, low-precision training, and test-time computation. A pretraining-loss law does not by itself predict every downstream capability, deployment cost, social effect, or future research breakthrough.

Scope and notation

The most common language-model notation is:

Scroll sideways for more →

The basic one-variable form is

$$L(x) = E + A x^{-\alpha},$$

where x is a scale variable, A > 0, and α > 0. Subtracting the fitted floor and taking logarithms gives

$$\log(L-E) = \log A - \alpha \log x.$$

The excess loss is therefore a straight line with slope −α on a log-log plot if the proposed power law is accurate. A small exponent still compounds over many orders of magnitude, but it also means that each fixed multiple of scale yields a smaller absolute improvement as the loss approaches the fitted floor.

The word "loss" matters. A smooth relationship for average next-token loss need not remain smooth after it is transformed into exact-match accuracy, pass rate, or another thresholded score. Likewise, two papers can report different exponents without disagreeing if they measure different outcomes or use different definitions of scale.

How scaling laws are estimated

Controlled scaling experiments

A controlled study trains a family of models while varying selected axes and holding the rest of the recipe as constant as practical. A useful record specifies:

• the model family and exact parameter-count convention;
• the training and evaluation distributions;
• the tokenizer or other data unit;
• optimizer hyperparameters, batch size, schedule, and stopping rule;
• numerical precision and whether routed parameters are active;
• the FLOP accounting convention;
• the metric and its uncertainty;
• which observations were used for fitting and which were held out for extrapolation.

The resulting curve is a conditional statement about that experimental system. Hyperparameters that are adequate at one scale can be inadequate at another, which can make an apparent scaling trend partly a tuning trend. This issue was central to later work reconciling language-model compute-allocation studies.^[3]

Interpolation and extrapolation

Interpolation evaluates a fit between observed scales. Extrapolation predicts beyond them and is the more demanding use of a scaling law. Rosenfeld et al. fitted a joint model of data size and model size and tested predictions on held-out larger configurations in ImageNet and WikiText-103 experiments. They found useful extrapolation in the studied regions, but also showed that fits can be unreliable when the observations remain near random-guess performance or do not yet identify the asymptotic regime.^[2]

Alabdulmohsin et al. argued that scaling estimators should be selected by extrapolation error rather than by in-sample fit alone. Their M4 estimator uses

$$\frac{\epsilon_x-\epsilon_{\infty}}{(\epsilon_0-\epsilon_x)^{\alpha}} = \beta x^c,$$

where ε₀ represents a random-guess level, ε∞ is a fitted limiting error, α ≥ 0, β > 0, and c < 0. It reduces to a power law with an additive floor when α = 0. The authors evaluated estimators on 90 tasks spanning image classification, neural machine translation, language modeling, and BIG-Bench, and reported that M4 more often extrapolated accurately in their benchmark.^[4]

Controlled and observational laws

Not every scaling analysis trains a complete model family. Ruan et al. studied about 100 publicly available language models from different families. Their observational model gave each family a different efficiency for converting reported compute into a low-dimensional capability representation. Several benchmark trends then followed smooth sigmoidal curves in that representation.^[5]

This can help analyze released models when controlled training is unaffordable, but it answers a different question. Public models differ in data, architecture, post-training, contamination risk, and reporting quality. An observational relationship can be predictive without establishing which intervention caused the change.

Early empirical work

Hestness et al. reported predictable power-law regions in 2017 across neural machine translation, language modeling, image classification, and speech recognition. The study varied training-set size and model capacity over multiple orders of magnitude. Both the target metric and fitted exponent differed by task: examples included cross entropy, classification error, and character error rate. Those exponents should not be ranked as if they were measurements of one common physical quantity.^[1]

The paper also distinguished a data-limited region from a model-capacity limit. Increasing data cannot improve a model that is too small to exploit it, while increasing model capacity yields little benefit when the available data are exhausted. This balance between resources became a central theme of later compute-optimal work.

Rosenfeld et al. subsequently proposed a single surface for generalization error as a function of both sample size and model size. The model approaches a power law when one resource is limiting and saturates when the other is fixed. Their held-out tests illustrated the practical purpose of scaling analysis: a set of smaller experiments can forecast a larger configuration, but only after the measured points enter a regime that constrains the fit.^[2]

The Kaplan language-model scaling laws

Kaplan et al. studied decoder-only Transformers trained on WebText2. Their main family ranged from 768 to 1.5 billion non-embedding parameters, used a 1,024-token context, and drew on a dataset containing about 22.9 billion tokens, with a separate test reserve. The unusually small lower endpoint refers to non-embedding parameters under the paper's convention, not the total stored weights.^[6]

One-variable fits

When one resource was treated as the active constraint, Kaplan et al. reported:

$$L(N) = \left(\frac{N_c}{N}\right)^{\alpha_N}, \qquad \alpha_N \approx 0.076, \qquad N_c \approx 8.8 \times 10^{13},$$ $$L(D) = \left(\frac{D_c}{D}\right)^{\alpha_D}, \qquad \alpha_D \approx 0.095, \qquad D_c \approx 5.4 \times 10^{13},$$

and, after adjusting to an estimate of minimum compute at small batch size,

$$L(C_{\min}) = \left(\frac{C_c}{C_{\min}}\right)^{\alpha_C}, \qquad \alpha_C \approx 0.050.$$

Here N excludes vocabulary and positional embeddings, D is measured in tokens, and the loss is measured on the paper's WebText2 evaluation distribution. The constants depend on tokenization and vocabulary and do not have a universal interpretation.^[6]

Within the Transformer shapes tested, loss depended much more strongly on non-embedding parameter count than on the particular combination of depth, width, and attention heads. The paper did not show that architecture is irrelevant outside that family or range.

Joint model and data dependence

Kaplan et al. fitted the joint form

$$L(N,D)= \left[ \left(\frac{N_c}{N}\right)^{\alpha_N/\alpha_D} +\frac{D_c}{D} \right]^{\alpha_D}.$$

The joint fit set the model exponent to 0.076 and the data exponent to 0.103, with scale constants of 6.4 × 10¹³ for model size and 1.8 × 10¹³ for data. The data exponent in this formula is therefore not the 0.095 exponent from the separate data-only fit. The functional form captures saturation: for a fixed dataset, additional parameters eventually give little improvement, and the reverse is also true.^[6]

Holding the degree of overfitting roughly constant in this model requires

$$D \propto N^{\alpha_N/\alpha_D} \approx N^{0.74}.$$

This was one expression of the paper's larger-model sample-efficiency result. It was not a recommendation to train every model at that ratio, because the paper's compute-efficient calculation used training dynamics and batch-size adjustments as well.

Compute-efficient allocation

For an unconstrained allocation of adjusted compute, Kaplan et al. found the empirical trends

$$N_{\mathrm{opt}} \propto C_{\min}^{0.73}, \qquad D_{\mathrm{opt}} \propto C_{\min}^{0.27}.$$

The corresponding prescription favored making the model substantially larger while increasing the number of optimization steps only slowly and stopping before full convergence. This result was conditional on the paper's optimizer, fixed-batch observations, critical-batch correction, model family, and compute accounting.^[6]

The study also fitted critical batch size as

$$B_{\mathrm{crit}}(L) \approx \frac{B_*}{L^{1/\alpha_B}}, \qquad B_* \approx 2\times10^8\ \text{tokens}, \qquad \alpha_B \approx 0.21.$$

The authors interpreted this as a time-compute tradeoff for their experiments: batches above the critical size gave diminishing reductions in optimization steps, while smaller batches used compute more efficiently. It is not a universal batch-size formula for all optimizers and objectives.

Relationship to GPT-3

The GPT-3 paper described a family culminating in a 175-billion-parameter model trained on about 300 billion tokens. Its authors cited the preceding language-model scaling evidence when selecting and interpreting the model family. GPT-3's design also involved a specific dataset mixture, architecture, context length, optimization recipe, and evaluation program, so its size should not be attributed to one scaling equation alone.^[7]

The Chinchilla compute-optimal study

Hoffmann et al. revisited how a fixed dense-language-model training budget should be divided between parameters and tokens. Their analysis used more than 400 models, from under 70 million to over 16 billion parameters. The main text describes runs from 5 billion to more than 400 billion tokens, while the abstract summarizes the upper range as 500 billion.^[8]

Three estimation approaches

The study used three distinct methods:

1. It fixed model sizes, varied training horizons, and selected the minimum loss at each compute budget.
2. It trained different model sizes at nine fixed compute budgets and fitted the minimum of each IsoFLOP profile.
3. It fitted a parametric loss function jointly to model size and training tokens.

Their estimated allocation exponents were:

Scroll sideways for more →

All three Chinchilla methods placed substantially more of the marginal budget into data than the Kaplan allocation. They supported approximate equal growth of parameters and tokens, not one exact exponent pair that applies independently of the training recipe.^[8]

Parametric loss and compute constraint

The third method used

$$L(N,D) = E + \frac{A}{N^\alpha} + \frac{B}{D^\beta},$$

with published rounded values

$$E=1.69,\quad A=406.4,\quad B=410.7,\quad \alpha=0.34,\quad \beta=0.28.$$

E is a fitted limiting term in this decomposition. Calling it the exact entropy of language would require assumptions and evidence beyond the empirical fit.

For a dense Transformer, the paper used the approximation

$$C \approx 6ND.$$

The factor accounts for a forward and backward training pass under a particular operation convention. It excludes or approximates some effects, and later work showed that whether the output head and attention operations are counted can materially affect small-model fits.^[3] Under this constraint, minimizing the parametric loss gives

$$N_{\mathrm{opt}} \propto C^{\beta/(\alpha+\beta)}, \qquad D_{\mathrm{opt}} \propto C^{\alpha/(\alpha+\beta)}.$$

These proportionalities are mathematical consequences of the assumed loss and compute forms. Their usefulness still depends on the empirical coefficients remaining valid.

Chinchilla and Gopher

The authors used the estimated frontier to train Chinchilla, a 70-billion-parameter model on 1.4 trillion tokens. It used the same reported training FLOPs as the 280-billion-parameter Gopher and approximately four times as many training tokens. The paper reported that Chinchilla outperformed Gopher and several other larger models across a broad evaluation suite, while individual tables retained task-level differences and evaluation-specific protocols.^[8]

The zero-shot common-sense table is:

Scroll sideways for more →

The PIQA result for Chinchilla is 81.8 percent, not 83.7 percent. The 83.7-percent value belongs to BoolQ. This distinction corrects a transcription error in an earlier version of this article.^[8]

The paper's MMLU abstract value is 67.5 percent, while a displayed five-shot table rounds the result to 67.6 percent. These are two presentations of the paper's result, not evidence for greater precision.

What "20 tokens per parameter" means

Several Chinchilla frontier estimates lie near 20 training tokens per parameter around the studied range. This became a convenient rule of thumb. It is not a universal constant. The three methods have different exponents, dataset quality and training recipe can move the frontier, and uncertainty grows when the fit is extrapolated far beyond observed runs.^[8][9]

Replication and reconciliation

Reconstruction of the Chinchilla parametric fit

Besiroglu et al. attempted to reproduce Chinchilla's third method from a digitized copy of a published plot because the original run table was not available. Their main fit used 240 reconstructed points after excluding five extreme configurations with fewer than 0.4 tokens per parameter. This was a reanalysis of displayed results, not a new training program of more than 400 models.^[9]

Their fitted function was

$$L(N,D)=1.8172+\frac{482.01}{N^{0.3478}}+ \frac{2085.43}{D^{0.3658}}.$$

They reported bootstrap standard errors and found that the published Chinchilla parameters did not fit the reconstructed points as well. The authors later identified two procedural issues. Rounded coefficients in the Chinchilla paper, especially the rounded data exponent, create a material prediction difference at large D. More precise values were available in the paper source. In addition, averaging rather than summing the Huber losses caused the original optimizer's stopping criterion to trigger early, affecting both the point fit and its unusually narrow bootstrap intervals.^[9]

The reanalysis did not overturn the broad conclusion that model size and data should grow at similar rates. Its fitted allocation was consistent with Chinchilla's first two methods. The point estimate implied about 25.6 tokens per parameter, and the authors' plotted uncertainty at training budgets of 10²⁶ FLOPs or more included ratios from about 4 to 40. This range is one reason to treat 20 as a local planning heuristic.

Controlled reconciliation of Kaplan and Chinchilla

Porian et al. performed more than 900 training runs on RefinedWeb and OpenWebText2. Their grid contained 16 decoder-only models from 5 million to 901 million parameters. They first reproduced a Kaplan-like setup, then changed one methodological factor at a time.^[3]

For RefinedWeb, the estimated model-size allocation exponent a changed as follows:

Scroll sideways for more →

The corresponding OpenWebText2 values were 0.864, 0.699, 0.603, 0.574, and 0.518. Counting the output head mattered most at small model sizes, where omitting it severely understated the actual matrix-operation cost. A fixed 3,000-step warmup also occupied too much of the compute-efficient training horizon for smaller models. Finally, using a single optimizer configuration across scales made small models relatively suboptimal.^[3]

Cosine learning-rate decay improved final loss, but the tuned constant-rate experiments still recovered a near-Chinchilla allocation exponent. The result therefore separates two questions: a schedule can improve attainable loss while not being necessary for a particular allocation trend. Porian et al. also cautioned that their largest models and budgets were much smaller than Hoffmann et al.'s and that their hyperparameter sweep focused on small scales.

Haegele et al. studied constant learning rates followed by cooldown periods as an alternative to a cosine schedule whose endpoint must be fixed in advance. In their experiments, a single trajectory could be cooled down at different horizons and reused in scaling analyses, reducing the need to retrain every model from the start for each duration. This improves the efficiency and flexibility of experimentation, but it does not make the resulting law independent of the schedule.^[10]

Allocation beyond a single pretraining run

Lifetime training and inference compute

Compute-optimal pretraining minimizes loss for a training budget. A deployed model can have a different objective because a larger model costs more on every inference. Sardana et al. considered the lifetime FLOP model

$$C_{\mathrm{life}} \approx 6ND_{\mathrm{train}} + 2ND_{\mathrm{inference}},$$

and minimized it for a fixed target loss and anticipated inference demand. The first term is a dense-model training approximation and the second approximates forward-pass inference. The calculation assumes that demand can be estimated and that models reaching the same fitted loss are interchangeable for the deployment objective.^[11]

The study trained 47 models from 150 million to 6 billion parameters over ratios from 10 to 10,000 tokens per parameter. It did not observe saturation from additional tokens within that tested range, but the authors explicitly noted the limited model scale, data, and number of runs. In one stated scenario, serving 2 trillion inference tokens at the quality of a 13-billion-parameter model made a more heavily trained 7-billion-parameter model 17 percent cheaper in lifetime FLOPs. The percentage belongs to those assumptions and is not a universal savings estimate.^[11]

Llama 3 as a documented example

Meta's Llama 3 report describes internal scaling experiments from 40 million to 16 billion parameters and from 6 × 10¹⁸ to 10²² FLOPs. For the largest run's budget, the report predicted approximately 402 billion parameters and 16.55 trillion tokens. The released 405-billion-parameter model was trained on 15.6 trillion text tokens with an estimated 3.8 × 10²⁵ training FLOPs, close to that internal prediction.^[12]

The same report says the 8-billion- and 70-billion-parameter models were deliberately trained longer than their pretraining compute optimum. At those fixed sizes, spending more training compute improved quality without increasing per-token inference compute. This is an economic and deployment tradeoff, not a contradiction of a pretraining frontier.

Precision-aware scaling

Numerical precision is another allocation axis. Kumar et al. trained 465 language models at precisions from 3 to 16 bits, with validation up to 1.7 billion parameters and 26 billion tokens. Their precision-aware model represents low-precision training as reducing an effective parameter count and adds a separate post-training quantization term to a Chinchilla-like loss.^[13]

For models pretrained in BF16 or higher precision and then weight-quantized, the measured degradation increased with training tokens and decreased with model size. The fitted dependence was approximately a power of the token-to-parameter ratio, along with an exponential precision term. In sufficiently overtrained and aggressively quantized configurations, the quantization damage could exceed the loss improvement from additional pretraining data. This is a result about the study's post-training quantization methods and precision range, not a claim that more data harms an unquantized model.

Data-constrained scaling

Compute-optimal analyses often assume a supply of fresh data. Muennighoff et al. instead trained more than 400 models from 10 million to 9 billion parameters in data-constrained settings, processing up to 900 billion tokens and, in extreme experiments, repeating a limited corpus for as many as 1,500 epochs.^[14]

At fixed compute in their setup, training for as many as four epochs over repeated data caused negligible loss change relative to the same number of unique tokens, and downstream differences were not statistically significant in the reported comparisons. Additional repetitions continued to help but with diminishing returns that approached zero. Their fitted law used "effective" data and parameter terms to represent the loss of value from repeated observations.

The four-epoch finding is not a general permission to repeat any corpus four times without cost. Memorization, duplication, data order, model capacity, objective, and distribution all matter. The study's conclusion is narrower: moderate repetition can remain useful when fresh tokens are unavailable, and repeated tokens should not be counted as having the same marginal value forever.

Data quality also changes scaling. Sorscher et al. showed that, with a sufficiently informative ranking, pruning lower-value examples could improve error scaling on CIFAR-10, SVHN, and ImageNet experiments. The benefit depended on the quality of the ranking method, which could itself require labels or expensive computation. Token or example count alone is therefore an incomplete description of a dataset.^[15]

Scaling beyond dense text models

Autoregressive modeling in other domains

Henighan et al. studied autoregressive generative modeling of text, images, video, image-text pairs, and mathematical sequences. Across these settings, loss often followed a power law plus a domain-specific constant over the observed range. The tasks used different tokenizations and outcomes, so the results support the portability of scaling analysis as a method rather than one exponent shared by all modalities.^[16]

Transfer learning

Hernandez et al. defined effective transferred data as the additional target-distribution data a same-size model trained from scratch would have needed to match a pretrained and fine-tuned model's loss. In a low-data text-to-Python setting they fitted

$$D_T = kD_F^{\alpha}N^{\beta},$$

Here, fine-tuning data is represented by D with subscript F, and N excludes embeddings. The experiments found that the exponent on model size was approximately 0.38 for two pretraining mixtures, while the exponent on fine-tuning data differed with the source distribution. The relationship held in the paper's low-data region; at high target-data levels, small pretrained models could show "ossification" and underperform from-scratch training.^[17]

This illustrates that transfer learning can have its own scaling law and boundary. A source distribution may multiply the value of scarce target data without replacing target data indefinitely.

Routed models

In a routed or mixture-of-experts model, total parameters and parameters active for one token are different axes. Clark et al. fitted a unified law using dense and routed language models over five orders of magnitude. Their effective parameter count summarized how the total expert capacity interacted with the active computation.^[18]

A routed model can therefore increase total representational capacity without activating every weight for every token. Comparing its total parameters with a dense model's active parameters, without also reporting active compute and routing setup, conflates distinct resources.

Vision scaling

Zhai et al. trained Vision Transformers from about 5 million to 2 billion parameters on datasets ranging from about 1 million to 3 billion images, using budgets from below one to more than 10,000 TPUv3 core-days. They identified performance-compute frontiers and saturation effects rather than one language-style token ratio.^[19]

Alabdulmohsin et al. later studied compute-optimal Vision Transformer shape. Their SoViT-400m/14 design balanced width and depth and was competitive with tested models more than twice its size at equivalent compute, with less than half their inference cost.^[20] Dehghani et al. scaled a Vision Transformer to 22 billion parameters and found improvements in several frozen-representation and transfer experiments.^[21] These papers show that architecture shape, image resolution, and downstream representation use are material variables in vision scaling.

Multimodal scaling

Aghajanyan et al. reported more than 250 experiments across seven modalities, with models from 8 million to 30 billion parameters and training sets from 5 billion to 100 billion tokens. Their mixed-modal law included an interaction term rather than treating each modality as an independent pile of interchangeable tokens. A 30-billion-parameter speech-text experiment served as a larger-scale validation within that setup.^[22]

Shukor et al. trained 457 native multimodal models with early- and late-fusion designs. Within models trained from scratch on their text, image-caption, and interleaved mixtures, early and late fusion had similar compute-optimal trends, and early fusion was somewhat stronger at low scale. The paper did not establish that early fusion is always preferable to a system assembled from separately pretrained components. Its sparse early-fusion experiments also required distinguishing active from total parameters.^[23]

Regime changes and broken laws

A single power law assumes one slope throughout the fitted range. Real learning curves can have random-guess regions, saturation, double descent, delayed improvements, or changes in data quality and optimization. Caballero et al. proposed the smoothly broken form

$$y = a + bx^{-c_0} \prod_{i=1}^{n} \left(1+\left(\frac{x}{d_i}\right)^{1/f_i}\right)^{-c_i f_i},$$

where each dᵢ locates a break, each cᵢ changes the log-log slope, and each fᵢ controls how sharp the transition is. With no breaks, the form reduces to a power law plus a constant. With breaks, it can represent inflections and nonmonotonic behavior that simpler monotone forms cannot.^[24]

The flexibility is useful when observations span a transition, but it creates two cautions. First, additional parameters can overfit a small set of runs. Second, a curve fitted before an unobserved break cannot reveal where that future break will occur. A visually straight segment is evidence about the measured interval, not proof that the next interval has the same slope.

From pretraining loss to capabilities

Reported emergence

Wei et al. defined an emergent ability as an ability that is not present in smaller models but is present in larger models. They catalogued benchmark curves in which accuracy remained near chance and then increased at larger reported scales. This is an operational description of evaluation results. It does not by itself establish that an internal cognitive capability underwent a physical phase transition.^[25]

Metric and resolution effects

Schaeffer et al. showed how an apparently sharp jump can be created when a smoothly changing underlying quantity is passed through a nonlinear or discontinuous metric. Exact-match accuracy gives no partial credit, while multiple-choice accuracy has coarse resolution in a finite test set. In studied tasks, continuous alternatives such as Brier score or token edit distance produced smoother trends.^[26]

This does not prove that every claimed emergence is an artifact. It shows that the metric and sample size are part of the scaling system and that a discontinuity in a chart need not imply a discontinuity in the model.

Du et al. controlled the corpus, tokenizer, and architecture while varying model and data size. Models at the same pretraining loss obtained similar downstream performance in their experiments. Some tasks remained at chance until loss crossed a task-specific threshold even under continuous metrics. The result supports using pretraining loss as a more comparable progress coordinate inside that controlled family, but it does not establish a threshold shared across families or datasets.^[27]

Ruan et al.'s observational analysis provides another view. After accounting for family-specific compute efficiency, several apparently emergent public-model trends were fitted by smooth sigmoids in a low-dimensional capability space. Because the models were not produced by one controlled experiment, the result is predictive evidence, not a causal account of the transition.^[5]

What loss cannot establish alone

Average next-token loss compresses performance across tokens and contexts into one number. Two models with similar loss can differ in calibration, rare-event behavior, long-context use, factual reliability, robustness, or a particular downstream task. Post-training can also move benchmark scores without changing the pretraining computation described by a Kaplan or Chinchilla law.

The metric can be socially incomplete even when its mathematical fit is excellent. Diaz and Madaio argue that an aggregate evaluation may reflect the priorities of an overrepresented group and fail to measure quality for other affected communities. That critique concerns what the dependent variable represents, not whether a power law happens to fit it.^[28]

Test-time compute

Pretraining scaling changes the model before deployment. Inference-time scaling spends more computation on a particular input through sampling, search, verification, revision, or longer reasoning. It introduces a new allocation problem among base-model quality, number and organization of attempts, verifier quality, and prompt difficulty.

Snell et al. studied MATH problems with PaLM 2 model families, using process-based verifiers and adaptive revision methods. Their compute-optimal policy allocated test-time work according to problem difficulty and was more than four times as efficient as a best-of-N baseline in the reported comparison. In a FLOP-matched experiment, a smaller model could outperform a model 14 times larger on problems where the smaller model already had a nontrivial success rate. On the hardest problems, where it rarely generated a useful starting answer, additional pretraining could remain more effective.^[29]

This result is conditional on the task, base models, verifier, and search procedure. It does not imply that an arbitrary small model can replace a much larger one by sampling more. Detailed algorithms and product-specific results belong to the separate test-time topic rather than to a general pretraining-loss law.

Data supply and generated data

Villalobos et al. estimated the stock and production flow of publicly available human-generated text and projected future training demand. Under continuation of the trends modeled in the paper, datasets would approach the effective public-text stock between 2026 and 2032, somewhat earlier if models were deliberately trained beyond their compute-optimal token counts.^[30]

This is a scenario range, not a guaranteed exhaustion date. The estimate depends on filtering, deduplication, multilingual inclusion, legal and access constraints, tokenization, data-efficiency progress, repetition, private data, and future data production. Reaching the stock estimate also does not mean every usable token has equal value.

Synthetic data can expand or transform a training distribution, but its provenance and filtering matter. Shumailov et al. studied recursive replacement, in which generated samples increasingly replace real samples across generations. Their theoretical and empirical examples showed progressive loss of distribution tails and eventual model collapse.^[31]

The experiment does not show that every use of generated data is harmful. Mixtures retaining real data, external verification, simulators with a grounded objective, rejection sampling, and targeted generation are different procedures. A 2025 author correction to the Nature article changed a symbol in its theoretical intuition; it did not reverse the paper's recursive-replacement result.

Why power laws can arise

Empirical fits describe what happened, while theories attempt to explain why. No single archived theory explains every observed neural scaling law.

Data-manifold account

Sharma and Kaplan analyzed approximation of functions on a d-dimensional manifold by piecewise-linear models. Under their assumptions, mean-squared error and the excess term of KL or cross-entropy loss scale as

$$L(N)-L_{\infty} \propto N^{-\alpha}, \qquad \alpha \approx \frac{4}{d}.$$

They tested aspects of this account in teacher-student networks and selected convolutional and language-model representations. The "dimension" is task- and representation-dependent, and the derivation assumes an approximation regime. It is a proposed mechanism, not a method for reading a universal intrinsic dimension directly from any fitted exponent.^[32]

Statistical-mechanics account

Bahri et al. analyzed learning with data spectra and manifold structure and identified four combinations of data and model limitations. In a variance-limited regime, error is controlled by finite-sample fluctuations. In a resolution-limited regime, the model cannot resolve increasingly fine structure. The predicted exponents depend on spectral decay and intrinsic-dimension assumptions.^[33]

These approaches help explain why broad scale-free regions can occur and why they eventually end. Noise floors, finite data, insufficient model capacity, optimization failure, or a shift to a new feature scale can all create plateaus or breaks.

Interpreting scaling results

A law is a conditional forecast

A scaling claim should be read as:

Under this model family, data construction, objective, optimizer, schedule, precision, metric, and counting convention, these observations are well described by this function over this range.

Changing one item can create a new law. Porian et al.'s reconciliation demonstrated this directly: output-head FLOPs, warmup, and scale-specific tuning changed the fitted compute allocation. Precision-aware work added bit width as another axis. Routed models require active and total capacity. Data pruning changes the value represented by a nominal example count.

Common sources of error

Scroll sideways for more →

FLOPs, hardware, energy, and money

Training FLOPs are an algorithmic operation estimate. Accelerator-hours additionally depend on hardware throughput, utilization, communication, memory traffic, sparsity, and precision. Electrical energy depends on average power over time and on datacenter overhead. Monetary cost further depends on ownership, depreciation, cloud pricing, labor, networking, and accounting choices.

Consequently,

$$\text{FLOPs} \neq \text{accelerator-hours} \neq \text{electricity} \neq \text{cash cost}.$$

The dense approximation shown above is useful for comparing certain training configurations, but it cannot justify a dollar or electricity figure without measured hardware and operational inputs. Unsupported conversions and investment forecasts should not be presented as consequences of a scaling law.

Diminishing returns and research progress

A positive power-law trend and diminishing marginal improvement can both be true. For the fitted form introduced in "Scope and notation," increasing x by a fixed factor continues to reduce the fitted excess loss, while the absolute reduction shrinks as L approaches E. Whether the improvement is worthwhile depends on the objective, not on the existence of the mathematical trend alone.

New architecture, data, optimization, precision, transfer, and test-time methods can shift a curve or define a new scaling axis. Such shifts do not prove unlimited continuation, and a plateau in one recipe does not prove that all scaling is over.

In a November 2025 interview, Ilya Sutskever described approximately 2020 through 2025 as an "age of scaling" centered on a predictable pretraining recipe. He argued that finite internet data and weak model generalization make renewed research more important. This was his interpretation of the field, with approximate dates and explicit uncertainty, not an empirical finding that every scaling relationship had stopped.^[34]

Practical use

A cautious workflow for a new project is:

1. Define the target outcome. Use validation loss for a pretraining forecast, or a deployment metric if that is the real objective.
2. Specify all scale variables and counting conventions before fitting.
3. Train a geometrically spaced set of models and data budgets large enough to expose both resource limits.
4. Tune optimization at multiple scales so that one part of the grid is not artificially weak.
5. Reserve larger observations for extrapolation rather than using every run to select the fit.
6. Fit several plausible forms, inspect residuals, and report parameter covariance or bootstrap intervals.
7. Test sensitivity to outliers, floors, schedule choices, data mixtures, and parameter definitions.
8. Refit after a material change in architecture, tokenizer, objective, data, precision, or routing.
9. Keep loss forecasts separate from capability, hardware, energy, and financial forecasts unless each conversion is independently modeled.

A scaling law is most useful as a reproducible planning instrument. Its credibility comes from transparent definitions, released observations, held-out predictions, and explicit failure conditions, not from the visual straightness of one log-log chart.

Reproducibility also requires preserving more than the final coefficients. A useful release includes the run-level model and data sizes, observed losses before smoothing, training checkpoints or sufficiently detailed curves, optimizer state and stopping criteria, excluded points, fitting code, initialization grid, residuals, and the exact objective minimized by the numerical solver. The Chinchilla reconstruction showed why this matters: plot digitization, coefficient rounding, reduction of per-example losses, and an optimizer stopping rule all affected what later investigators could recover.^[9] The Porian experiments likewise showed that an apparently minor counting or warmup convention could move the fitted allocation exponent.^[3] Publishing these details lets other investigators separate a disagreement about data from a disagreement about definitions or numerical procedures.

Scaling studies should also disclose unsuccessful or dominated configurations. Removing runs only because they do not lie near the preferred frontier can make uncertainty appear smaller and can conceal a regime in which optimization failed. Exclusions may be justified, but the criterion should be specified before or alongside the fit and the sensitivity to retaining those observations should be reported.

Timeline

Scroll sideways for more →

See also

• Deep learning
• Large language model
• Scaling Laws for Neural Language Models
• Chinchilla scaling laws
• Pre-training
• Test-time compute
• Quantization

References

1. ^Joel Hestness et al. "Deep Learning Scaling is Predictable, Empirically." 2017. arXiv:1712.00409.
2. ^Jonathan S. Rosenfeld et al. "A Constructive Prediction of the Generalization Error Across Scales." ICLR, 2020. OpenReview.
3. ^Thaddaeus Porian et al. "Resolving Discrepancies in Compute-Optimal Scaling of Language Models." NeurIPS, 2024. arXiv:2406.19146.
4. ^Ibrahim Alabdulmohsin, Behnam Neyshabur, and Xiaohua Zhai. "Revisiting Neural Scaling Laws in Language and Vision." NeurIPS, 2022. NeurIPS proceedings.
5. ^Yangjun Ruan, Chris J. Maddison, and Tatsunori Hashimoto. "Observational Scaling Laws and the Predictability of Language Model Performance." NeurIPS, 2024. NeurIPS proceedings.
6. ^Jared Kaplan et al. "Scaling Laws for Neural Language Models." 2020. arXiv:2001.08361.
7. ^Tom B. Brown et al. "Language Models are Few-Shot Learners." NeurIPS, 2020. NeurIPS proceedings.
8. ^Jordan Hoffmann et al. "Training Compute-Optimal Large Language Models." NeurIPS, 2022. arXiv:2203.15556.
9. ^Tamay Besiroglu et al. "Chinchilla Scaling: A Replication Attempt." 2024. arXiv:2404.10102.
10. ^Alexander Haegele et al. "Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations." NeurIPS, 2024. NeurIPS proceedings.
11. ^Nikhil Sardana et al. "Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws." ICML, 2024. PMLR.
12. ^Aaron Grattafiori et al. "The Llama 3 Herd of Models." 2024. arXiv:2407.21783.
13. ^Tanishq Kumar et al. "Scaling Laws for Precision." ICLR oral, 2025. OpenReview.
14. ^Niklas Muennighoff et al. "Scaling Data-Constrained Language Models." NeurIPS, 2023. arXiv:2305.16264.
15. ^Ozan Sorscher et al. "Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning." NeurIPS, 2022. arXiv:2206.14486.
16. ^Tom Henighan et al. "Scaling Laws for Autoregressive Generative Modeling." 2020. arXiv:2010.14701.
17. ^Danny Hernandez et al. "Scaling Laws for Transfer." 2021. arXiv:2102.01293.
18. ^Aidan Clark et al. "Unified Scaling Laws for Routed Language Models." ICML, 2022. PMLR.
19. ^Xiaohua Zhai et al. "Scaling Vision Transformers." CVPR, 2022. CVF Open Access.
20. ^Ibrahim Alabdulmohsin et al. "Getting ViT in Shape: Scaling Laws for Compute-Optimal Model Design." NeurIPS, 2023. arXiv:2305.13035.
21. ^Mostafa Dehghani et al. "Scaling Vision Transformers to 22 Billion Parameters." ICML, 2023. PMLR.
22. ^Armen Aghajanyan et al. "Scaling Laws for Generative Mixed-Modal Language Models." 2023. arXiv:2301.03728.
23. ^Mustafa Shukor et al. "Scaling Laws for Native Multimodal Models." ICCV, 2025. CVF Open Access.
24. ^Ethan Caballero et al. "Broken Neural Scaling Laws." ICLR, 2023. arXiv:2210.14891.
25. ^Jason Wei et al. "Emergent Abilities of Large Language Models." TMLR, 2022. OpenReview.
26. ^Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. "Are Emergent Abilities of Large Language Models a Mirage?" NeurIPS, 2023. arXiv:2304.15004.
27. ^Zhengxiao Du et al. "Understanding Emergent Abilities of Language Models from the Loss Perspective." NeurIPS, 2024. NeurIPS proceedings.
28. ^Fernando Diaz and Michael Madaio. "Scaling Laws Do Not Scale." AAAI/ACM AIES, 2024. arXiv:2307.03201.
29. ^Charlie Snell et al. "Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters." ICLR oral, 2025. arXiv:2408.03314.
30. ^Pablo Villalobos et al. "Will We Run Out of Data? Limits of LLM Scaling Based on Human-Generated Data." 2024. arXiv:2211.04325.
31. ^Ilia Shumailov et al. "AI Models Collapse When Trained on Recursively Generated Data." Nature, 2024. DOI:10.1038/s41586-024-07566-y.
32. ^Utkarsh Sharma and Jared Kaplan. "Scaling Laws from the Data Manifold Dimension." JMLR, 2022. JMLR.
33. ^Yasaman Bahri et al. "Explaining Neural Scaling Laws." PNAS, 2024. DOI:10.1073/pnas.2311878121.
34. ^Dwarkesh Patel. "Ilya Sutskever: We're moving from the age of scaling to the age of research." November 25, 2025. Dwarkesh Podcast transcript.