# Scaling

> Source: https://aiwiki.ai/wiki/scaling
> Updated: 2026-06-28
> Categories: Data & Datasets, Machine Learning
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution to "AI Wiki (aiwiki.ai)".

*See also: [Machine learning terms](/wiki/machine_learning_terms), [scaling laws](/wiki/scaling_laws), [test-time compute](/wiki/test_time_compute)*

In [machine learning](/wiki/machine_learning) and artificial intelligence, **scaling** is an umbrella term for several distinct practices: (1) **feature scaling**, the data-preprocessing step that rescales input features to a common range so no single feature dominates; (2) **neural scaling laws**, the empirical finding that model loss falls as a smooth power law in the number of parameters, the volume of training data, and the amount of compute; (3) **model (architecture) scaling**, adjusting a network's depth, width, and input resolution; and (4) **test-time (inference-time) scaling**, spending extra compute during inference to improve answers. The most-cited result in the field is the [Chinchilla](/wiki/chinchilla) finding that compute-optimal training requires roughly 20 training tokens per model parameter, meaning many large models had been significantly undertrained [2].

This article covers all of these uses, starting with feature scaling in data preprocessing and continuing through scaling laws, model architecture scaling, and test-time compute scaling. Where appropriate it links to dedicated AI Wiki pages such as [scaling laws](/wiki/scaling_laws) and [Chinchilla](/wiki/chinchilla) rather than duplicating them.

## Why Feature Scaling Matters

Feature scaling is a critical preprocessing step because many machine learning algorithms are sensitive to the relative magnitudes of input features [8]. Without scaling, a feature measured in thousands (such as annual income) can dominate a feature measured in single digits (such as number of bedrooms), leading to poor model behavior. The [scikit-learn](/wiki/scikit-learn) documentation states the problem directly: "If a feature has a variance that is orders of magnitude larger than others, it might dominate the objective function and make the estimator unable to learn from other features correctly as expected" [5]. The primary reasons scaling matters are outlined below.

### Faster Convergence in Gradient Descent

Algorithms that use [gradient descent](/wiki/gradient_descent) for optimization, including [linear regression](/wiki/linear_regression), [logistic regression](/wiki/logistic_regression), and [neural networks](/wiki/neural_network), converge much faster when features are on a similar scale. When feature magnitudes differ widely, the loss surface becomes elongated, causing the optimizer to zigzag slowly toward the minimum. Scaling produces a more spherical loss surface, allowing the optimizer to take more direct steps.

### Fair Treatment of Features in Distance-Based Algorithms

Algorithms that compute distances between data points, such as [k-nearest neighbors](/wiki/k_nearest_neighbors), [k-means clustering](/wiki/k-means), and [support vector machines](/wiki/support_vector_machine_svm), are directly affected by feature magnitudes. A feature with a range of 0 to 100,000 will dominate Euclidean distance calculations over a feature with a range of 0 to 1. Scaling ensures every feature contributes proportionally to distance computations.

### Correct Regularization Behavior

Regularization techniques such as [L1 regularization](/wiki/l1_regularization) and [L2 regularization](/wiki/l2_regularization) add penalty terms based on the magnitudes of model weights. If features are on different scales, the corresponding weights will also be on different scales, and the regularization penalty will be applied unevenly. For instance, a feature with a large numeric range will have a small weight, receiving a disproportionately small penalty. Scaling ensures that the regularization parameter applies uniformly across all features, so feature selection and weight shrinkage reflect genuine predictive signal rather than differences in units.

### Improved Numeric Stability

Scaling helps prevent issues related to floating-point arithmetic. Very large or very small feature values can lead to numerical overflow or underflow during matrix operations, gradient computations, and activation functions. Keeping values in a moderate range improves the stability of training.

## Feature Scaling Methods

Several techniques are commonly used to scale features. Each has its own strengths and is suited to different situations. The table below summarizes the most widely used methods.

| Method | Formula | Output Range | Handles Outliers | Best Used When |
|---|---|---|---|---|
| Min-Max Scaling | x' = (x - x_min) / (x_max - x_min) | [0, 1] | No | Bounded data with no extreme outliers |
| Standardization (Z-score) | x' = (x - mean) / std | Unbounded (mean=0, std=1) | Partially | Data is roughly Gaussian; most general-purpose choice |
| Max-Abs Scaling | x' = x / abs(x_max) | [-1, 1] | No | Sparse data (preserves zero entries) |
| Robust Scaling | x' = (x - median) / IQR | Unbounded | Yes | Data with significant outliers |
| Unit Vector (L2 Norm) | x' = x / \|\|x\|\| | Unit length | No | When direction matters more than magnitude |

### Min-Max Scaling

Min-max scaling, also called [normalization](/wiki/normalization), transforms each feature to a fixed range, typically [0, 1]. The formula is:

x' = (x - x_min) / (x_max - x_min)

This method preserves the original distribution shape and produces bounded outputs. It works well when features have a known, finite range and the data contains no extreme outliers. However, a single outlier can compress the remaining values into a narrow band, reducing the effective resolution of the feature. In scikit-learn, this is implemented as `MinMaxScaler` [5].

### Standardization (Z-Score Normalization)

Standardization, also known as [z-score normalization](/wiki/z-score_normalization), centers each feature to have a mean of zero and scales it to have a standard deviation of one:

x' = (x - mean) / std

This is the most widely used scaling method and is suitable for algorithms that assume input features follow a Gaussian distribution, such as [support vector machines](/wiki/support_vector_machine_svm) and [logistic regression](/wiki/logistic_regression). The scikit-learn maintainers note that "standardization of datasets is a common requirement for many machine learning estimators implemented in scikit-learn; they might behave badly if the individual features do not more or less look like standard normally distributed data: Gaussian with zero mean and unit variance" [5]. Unlike min-max scaling, standardization does not bound the output to a specific range, which means outliers are less likely to distort the scale of normal values. In scikit-learn, this is implemented as `StandardScaler` [5].

### Max-Abs Scaling

Max-abs scaling divides each feature by its maximum absolute value:

x' = x / |x_max|

This produces values in the range [-1, 1] without shifting or centering the data. Its key advantage is that it preserves sparsity: zero values remain zero after transformation. This makes it particularly useful for sparse datasets, such as those produced by [bag of words](/wiki/bag_of_words) representations or [TF-IDF](/wiki/tf_idf) vectorization. In scikit-learn, this is implemented as `MaxAbsScaler` [5].

### Robust Scaling

Robust scaling uses the median and interquartile range (IQR) instead of the mean and standard deviation:

x' = (x - median) / IQR

Because the median and IQR are robust statistics, this method is much less sensitive to outliers than standardization or min-max scaling. It is the preferred choice when the dataset contains significant outliers that would otherwise distort the scaling. In scikit-learn, this is implemented as `RobustScaler` [5].

### Mean Normalization

Mean normalization centers each feature by subtracting the mean and dividing by the range:

x' = (x - mean) / (x_max - x_min)

This produces values centered around zero, typically in the range [-1, 1]. It combines aspects of both min-max scaling and standardization.

## Which Models Need Feature Scaling

Not all machine learning models are equally sensitive to feature scale. The table below summarizes which categories of models benefit from scaling and which do not.

| Model Category | Examples | Scaling Needed? | Reason |
|---|---|---|---|
| Linear models | [Linear regression](/wiki/linear_regression), [logistic regression](/wiki/logistic_regression) | Yes | Weights are directly influenced by feature magnitudes; regularization requires uniform scales |
| Distance-based models | [k-NN](/wiki/k_nearest_neighbors), [k-means](/wiki/k-means), [SVM](/wiki/support_vector_machine_svm) | Yes | Distance metrics are dominated by large-scale features |
| Neural networks | [MLP](/wiki/feedforward_neural_network_ffn), [CNN](/wiki/convolutional_neural_network), [Transformer](/wiki/transformer) | Yes | Gradient-based optimization converges faster with scaled inputs |
| Gradient boosting | [XGBoost](/wiki/gradient_boosting), LightGBM, CatBoost | No | Splits are based on thresholds; feature order is invariant to scaling |
| Tree-based models | [Decision tree](/wiki/decision_tree), [random forest](/wiki/random_forest) | No | Splits compare values within each feature independently; relative ordering is unaffected by scaling |
| Naive Bayes | Gaussian NB, Multinomial NB | No | Parameters are estimated per feature independently |

Tree-based models do not need scaling because they make decisions by finding threshold values that best split the data at each node. These splits depend only on the relative ordering of values within each feature, not on their absolute magnitudes. Whether a feature ranges from 0 to 1 or from 0 to 1,000,000, the same split points produce the same partitioning of data.

## How Do You Apply Feature Scaling in Scikit-Learn?

The [scikit-learn](/wiki/scikit-learn) library provides a consistent API for feature scaling through its `preprocessing` module [5]. All scalers follow the fit-transform pattern:

```python
from sklearn.preprocessing import StandardScaler, MinMaxScaler, RobustScaler, MaxAbsScaler
from sklearn.model_selection import train_test_split

# Split data first to prevent data leakage
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)

# Fit on training data only, then transform both sets
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)
```

A critical best practice is to fit the scaler only on the training data and then use the fitted scaler to transform the test data. Fitting on the full dataset (including test data) causes data leakage, because the scaler parameters (mean, standard deviation, min, max) would incorporate information from the test set. In production pipelines, the scaler should be serialized alongside the model so that new data receives the same transformation.

## What Are Neural Scaling Laws?

Beyond data preprocessing, "scaling" in AI also refers to how model performance improves as key resources (parameters, data, and compute) increase. Researchers have discovered that neural network loss follows predictable power-law relationships with these resources, a finding with profound implications for how large models are trained [9]. This section summarizes the central results; the dedicated [scaling laws](/wiki/scaling_laws) page covers the topic in more depth.

### Kaplan et al. (2020)

In January 2020, researchers at [OpenAI](/wiki/openai) published "Scaling Laws for Neural Language Models," establishing that the cross-entropy loss of language models decreases as a smooth power law with respect to three factors: the number of model parameters (N), the size of the training dataset (D), and the amount of training compute (C) [1]. The paper's opening claim is precise: "The loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude" [1]. The relationship takes the form:

L(x) = (x_0 / x)^alpha

where x represents any of the three factors and alpha is a scaling exponent. For model parameters, alpha was found to be approximately 0.076, meaning that each tenfold increase in parameters yields a consistent reduction in loss.

A key finding was that, for a fixed compute budget, optimal performance comes from training a very large model on a relatively modest amount of data and stopping training well before convergence. As the authors put it, "larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data and stopping significantly before convergence" [1]. Kaplan et al. proposed that optimal model size should scale as N_opt proportional to C^0.73, while dataset size should scale only weakly with compute [1]. This implied that scaling up model parameters was more important than scaling up training data.

### How Do Chinchilla Scaling Laws Differ from Kaplan?

In 2022, a team at Google [DeepMind](/wiki/deepmind) led by Jordan Hoffmann challenged the Kaplan findings with the paper "Training Compute-Optimal Large Language Models" [2]. Their key contribution was the concept of **compute-optimal training**: for a given compute budget C, the model size N and dataset size D should be scaled in equal proportions. The abstract states the rule plainly: "for every doubling of model size the number of training tokens should also be doubled" [2]. The team fit this relationship across more than 400 models ranging from 70 million to over 16 billion parameters, trained on 5 to 500 billion tokens [2].

The Chinchilla loss model is:

L(N, D) = A/N^alpha + B/D^beta + L_0

where A = 406.4, B = 410.7, alpha = 0.34, beta = 0.28, and L_0 = 1.69 (the irreducible loss, representing the inherent entropy of natural language) [2]. The total compute is approximated as C = 6ND (six floating-point operations per parameter per token).

The critical finding was that both N and D should scale proportionally with compute: N_opt proportional to C^0.5 and D_opt proportional to C^0.5. In practical terms, this means approximately 20 training tokens per parameter for optimal compute efficiency. The [Chinchilla](/wiki/chinchilla) model (70B parameters, 1.4 trillion tokens) outperformed the much larger [Gopher](/wiki/gopher) (280B parameters, 300 billion tokens) while using the same compute budget, demonstrating that many existing large models were significantly undertrained [2]. Chinchilla reached a state-of-the-art average accuracy of 67.5% on the [MMLU](/wiki/mmlu) benchmark, which the authors described as "greater than a 7% improvement over Gopher" [2].

### Reconciling the Two Approaches

The discrepancy between Kaplan and Chinchilla scaling laws stems from methodological differences. Kaplan et al. excluded embedding parameters from their parameter counts and used a fixed learning rate schedule that did not fully account for differences in training duration. When these differences are corrected, the two sets of results become more consistent. A 2024 replication by Besiroglu et al. scrutinized the original Chinchilla parametric fit and reported revised coefficient estimates, while broadly confirming the headline 20-tokens-per-parameter guidance [2]. The Chinchilla framework has become the standard reference for compute-optimal training and has directly influenced the design of models such as [LLaMA](/wiki/llama), which was explicitly trained following compute-optimal principles with significantly more data than earlier models of similar size.

### Impact on Industry

Neural scaling laws have shaped the strategies of major AI laboratories. Rather than training the largest possible model, teams now balance model size with data quantity and training duration to maximize performance per unit of compute. This has led to a generation of models ([LLaMA](/wiki/llama), [Mistral](/wiki/mistral), [Gemma](/wiki/gemma)) that achieve strong performance at smaller sizes by training on much larger datasets than their predecessors. Later work, including Muennighoff et al. (2024) on "Scaling Data-Constrained Language Models," extended the analysis to settings where high-quality data, rather than compute, is the binding constraint [7].

## What Is Model Scaling (Architecture Scaling)?

Model scaling refers to strategies for adjusting the dimensions of a neural network architecture to improve performance. For convolutional neural networks, the three primary scaling dimensions are:

- **Depth:** the number of layers in the network
- **Width:** the number of channels or neurons per layer
- **Resolution:** the size of the input image or sequence length

### EfficientNet and Compound Scaling

The 2019 paper "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks" by Mingxing Tan and Quoc Le introduced **compound scaling**, which scales all three dimensions simultaneously using a single compound coefficient [3]. Instead of scaling depth, width, or resolution independently, the method uses the relationships:

- depth: d = alpha^phi
- width: w = beta^phi
- resolution: r = gamma^phi

under the constraint alpha * beta^2 * gamma^2 approximately equals 2, where phi is the compound coefficient. The optimal base values were found to be alpha = 1.2, beta = 1.1, and gamma = 1.15.

The resulting [EfficientNet](/wiki/efficientnet) family (B0 through B7) demonstrated that compound scaling produces consistently better accuracy per unit of compute compared to scaling any single dimension. According to the paper, EfficientNet-B7 "achieves state-of-the-art 84.4% top-1 / 97.1% top-5 accuracy on ImageNet, while being 8.4x smaller and 6.1x faster on inference than the best existing ConvNet" [3]. EfficientNet V2, published in 2021, further improved training speed by incorporating progressive learning and adaptive regularization.

Compound scaling principles have since been extended beyond image classification on [ImageNet](/wiki/imagenet) to object detection, segmentation, and other vision tasks.

## What Is Test-Time Compute Scaling?

Test-time compute scaling (also called inference-time scaling) is a more recent paradigm in which additional computation is applied during inference, rather than during training, to improve model outputs [10]. This approach has become a major research direction since 2024. The dedicated [test-time compute](/wiki/test_time_compute) page covers the methods in more detail.

### The Core Idea

Traditional scaling focuses on making models larger or training them on more data. Test-time scaling takes a different approach: given a fixed pretrained model, spend more compute at inference time to produce better answers. This is analogous to a human thinking longer and more carefully about a difficult problem before answering.

### OpenAI o1 and Reasoning Models

OpenAI's [o1](/wiki/openai_o1) model, released in September 2024, demonstrated that test-time scaling could produce dramatic improvements in reasoning tasks. The model uses an extended internal [chain of thought](/wiki/chain_of_thought), generated through [reinforcement learning](/wiki/reinforcement_learning), to "think" through problems step by step before producing a final answer [6]. Performance scales smoothly with the amount of test-time compute allocated: more thinking time yields better results on mathematics, coding, and scientific reasoning benchmarks [6].

The [o3](/wiki/openai_o3) model, announced on December 20, 2024, pushed this further by achieving 87.5% on the ARC-AGI benchmark in a high-compute configuration. OpenAI released o3-mini on January 31, 2025, as a more cost-efficient variant optimized for strong reasoning at lower compute budgets.

### Research Directions

Several approaches to test-time scaling have been explored:

| Approach | Description | Example |
|---|---|---|
| Chain-of-thought | Model generates intermediate reasoning steps before the final answer | OpenAI o1, o3 |
| Best-of-N sampling | Generate multiple candidate answers and select the best one using a verifier | [AlphaCode](/wiki/alphacode) |
| Tree search | Explore a tree of reasoning paths and select the most promising branches | [AlphaProof](/wiki/alphaproof) |
| Budget forcing | Control the length of reasoning chains to trade off between compute cost and accuracy | s1 model (2025) |
| Self-refinement | Model iteratively critiques and improves its own output | Various research papers |

Research from Snell et al. (2024) showed that scaling inference compute with the right strategies can be more effective than scaling model parameters [4]. The authors report that a compute-optimal strategy can "improve the efficiency of test-time compute scaling by more than 4x compared to a best-of-N baseline," and that on suitable problems "test-time compute can be used to outperform a 14x larger model" [4]. The budget-forcing technique in the table comes from "s1: Simple test-time scaling" (Muennighoff et al., 2025), which controls compute by appending the token "Wait" to lengthen the model's thinking or by forcefully terminating it, reproducing OpenAI's test-time scaling curves using supervised fine-tuning on just 1,000 examples [11].

### Infrastructure Implications

Test-time scaling has significant cost implications. Reasoning models such as o1 and [DeepSeek-R1](/wiki/deepseek_r1) generate far more tokens than standard models because their hidden chain-of-thought adds a large number of intermediate tokens even on simple prompts. This raises the per-query cost of inference even when the per-token price is low: DeepSeek-R1 was hosted at roughly $0.55 per million input tokens and $2.19 per million output tokens at launch, but the volume of reasoning tokens it emits can make total query costs substantially higher than for a non-reasoning model. These dynamics have prompted research into more efficient inference strategies, including distillation of reasoning capabilities into smaller models.

## Explain Like I'm 5 (ELI5)

Imagine you have a recipe that calls for 2 cups of flour and 1 teaspoon of salt. If you tried to compare those amounts directly, the flour would seem way more important because 2 cups is a bigger number than 1 teaspoon. But they are just measured in different units. Feature scaling is like converting everything to the same unit so the computer can fairly compare all the ingredients.

Scaling laws are a different idea. They are like noticing that if you give a student twice as many textbooks and twice as much study time, their test scores improve by a predictable amount. Scientists have found that AI models follow similar patterns: making them bigger or giving them more data improves their performance in a very predictable way.

Test-time scaling is like giving a student extra time on an exam. The student already knows what they know, but with more time to think carefully, check their work, and try different approaches, they get better scores.

## References

1. Kaplan, J., McCandlish, S., Henighan, T., et al. (2020). "Scaling Laws for Neural Language Models." arXiv:2001.08361.
2. Hoffmann, J., Borgeaud, S., Mensch, A., et al. (2022). "Training Compute-Optimal Large Language Models." Proceedings of NeurIPS 2022. arXiv:2203.15556.
3. Tan, M. and Le, Q. (2019). "EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks." Proceedings of ICML 2019. arXiv:1905.11946.
4. Snell, C., Lee, J., Xu, K., and Kumar, A. (2024). "Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters." arXiv:2408.03314.
5. Pedregosa, F., Varoquaux, G., Gramfort, A., et al. (2011). "Scikit-learn: Machine Learning in Python." Journal of Machine Learning Research, 12, pp. 2825-2830. See also the scikit-learn "Preprocessing data" user guide.
6. OpenAI. (2024). "Learning to Reason with LLMs." OpenAI Blog.
7. Muennighoff, N., Rush, A., Barak, B., et al. (2024). "Scaling Data-Constrained Language Models." Proceedings of NeurIPS 2023.
8. Wikipedia contributors. "Feature scaling." Wikipedia, The Free Encyclopedia.
9. Wikipedia contributors. "Neural scaling law." Wikipedia, The Free Encyclopedia.
10. Zhang, Q., et al. (2025). "A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?" arXiv:2503.24235.
11. Muennighoff, N., Yang, Z., Shi, W., et al. (2025). "s1: Simple test-time scaling." arXiv:2501.19393; Proceedings of EMNLP 2025.