Batch Normalization

Batch normalization (BatchNorm or BN) is a normalization method used inside some neural networks. During training, it standardizes each selected activation using a mean and variance calculated from a mini-batch, then applies a learned scale and offset. A conventional BN layer uses fixed estimates of the training population's statistics during inference, so its behavior differs between training and evaluation.^[1]

Sergey Ioffe and Christian Szegedy introduced batch normalization at ICML in 2015. In their Inception experiment, a batch-normalized model reached the baseline's 72.2% validation accuracy in 14 times fewer training steps. A six-model ensemble obtained 4.9% top-5 error on the ImageNet validation set and 4.82% on the test server.^[1] BN was subsequently used in influential convolutional neural networks, including ResNet.^[20] Its batch dependence can also be a liability: output for one training example depends on the other examples grouped with it, small or correlated normalization batches can produce poor statistics, and stored statistics can mismatch the data seen at inference.^[5][6]

Definition

For one scalar feature, let a normalization batch be (B={x_1,\ldots,x_m}). The training transform is:

$$\mu_B=\frac{1}{m}\sum_{i=1}^{m}x_i$$ $$\sigma_B^2=\frac{1}{m}\sum_{i=1}^{m}(x_i-\mu_B)^2$$ $$\hat{x}_i=\frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}}, \qquad y_i=\gamma\hat{x}_i+\beta$$

Here, (\epsilon>0) limits numerical instability, while (\gamma) and (\beta) are learned by backpropagation with the network's other parameters. The original algorithm uses the population-form mini-batch variance with divisor (m), not the sample variance with divisor (m-1).^[1] ONNX BatchNormalization version 15 likewise defines the training variance as the mean squared deviation over the reduction set.^[17]

Scroll sideways for more →

The intermediate values (\hat{x}_i) have mean zero over the batch. Their variance is (\sigma_B^2/(\sigma_B^2+\epsilon)), which approaches one when (\epsilon) is small relative to the observed variance. The final output (y_i) does not generally have zero mean or unit variance because (\gamma) and (\beta) can change both.^[1] This distinction is important: BN normalizes and then restores an affine degree of freedom rather than forcing every layer output to stay standardized.

The affine parameters preserve representational capacity. For a fixed mean and variance, setting (\gamma=\sqrt{\sigma_B^2+\epsilon}) and (\beta=\mu_B) reproduces the identity transform. When an affine or convolutional operation is immediately followed by BN, its bias is usually redundant because batch mean subtraction cancels a constant offset and BN's (\beta) supplies an offset after normalization.^[1]

Normalization axes

The phrase "batch size" can hide the actual number of values used for each statistic. For a dense layer represented as (N\times C), BN normally computes one mean and variance for each of the (C) features across the (N) examples. For an image tensor in channels-first form (N\times C\times H\times W), spatial BN computes statistics separately for each channel over (N), (H), and (W). Each channel therefore uses (N H W) values and has one (\gamma_c) and one (\beta_c).^[1][14][17]

Scroll sideways for more →

This is not full whitening. BN treats scalar features or channels separately and does not generally remove correlations between different channels. Ioffe and Szegedy chose per-feature normalization in part because estimating and differentiating a full covariance transform for every mini-batch would be more expensive and can be ill-conditioned when the feature dimension is large relative to the batch.^[1]

Placement in a block

The original paper inserted BN after an affine or convolutional transform and before the nonlinearity, for example convolution, BN, then ReLU.^[1] ResNet's original residual blocks also used BN after each convolution and before activation, with an additional activation after the residual addition.^[20] Other architectures may use a different order. Placement is therefore an architectural choice that should be inherited from the tested model design, not treated as a universal rule.

Training and inference

Training mode

In conventional training mode, every forward pass calculates (\mu_B) and (\sigma_B^2) from the current normalization batch. Gradients pass through the mean and variance calculations. Consequently, an example's normalized activation depends on other examples in the same batch, and changing batch membership can change both the forward output and the gradient.^[1][5]

Most implementations also update running estimates of mean and variance during these forward passes. Those state updates are distinct from gradient updates to (\gamma) and (\beta). The stored statistics summarize activations produced by many earlier parameter states, so they can lag when the network changes rapidly. Wu and Johnson showed that exponential moving averages can be inaccurate in some training regimes and described recomputing population statistics from a fixed model state, which they call PreciseBN.^[5]

Evaluation mode

At ordinary inference, BN uses fixed estimated population statistics:

$$y=\gamma\frac{x-\mu_{\mathrm{pop}}} {\sqrt{\sigma_{\mathrm{pop}}^2+\epsilon}}+\beta$$

This makes each prediction independent of the other examples submitted at the same time and removes mini-batch randomness.^[1] It also creates a train-evaluation discrepancy because training used current mini-batch statistics. The discrepancy may be small under independently sampled, adequately sized batches and similar train and test distributions, but it can become material with small batches, rapid feature changes, fine-tuning, or distribution shift.^[5][6]

Once (\mu_{\mathrm{pop}}), (\sigma_{\mathrm{pop}}^2), (\gamma), and (\beta) are fixed, BN is an affine transform. For a preceding convolution with weights (W) and bias (b), define:

$$a=\frac{\gamma}{\sqrt{\sigma_{\mathrm{pop}}^2+\epsilon}}$$

The convolution and BN can be folded into (W'=aW) and (b'=a(b-\mu_{\mathrm{pop}})+\beta), with channel-wise broadcasting. This removes a separate BN operation without changing evaluation output, subject to the deployment system's numerical rules. Folding is an inference optimization; fusing a frozen affine transform before continued optimization can change training dynamics.^[1][5]

Variance estimators and momentum conventions

Framework details are not identical. In PyTorch 2.9, BatchNorm2d uses a biased variance estimator for the training forward pass but stores an unbiased variance estimate in its moving average. Its default running-statistics update is:

$$s_{\mathrm{new}}=(1-\text{momentum})s_{\mathrm{old}} +\text{momentum}\,s_{\mathrm{batch}}$$

with default momentum=0.1.^[14] TensorFlow 2.16 uses the opposite coefficient naming:

$$s_{\mathrm{new}}=\text{momentum}\,s_{\mathrm{old}} +(1-\text{momentum})s_{\mathrm{batch}}$$

with default momentum=0.99 and default epsilon=0.001.^[16] Thus a momentum value cannot be copied between frameworks without checking its definition. ONNX version 15 follows the TensorFlow-style coefficient convention and defaults to momentum 0.9 and epsilon (10^{-5}).^[17]

Why batch normalization can help

BN often improves optimization, but the literature does not support one settled mechanism that explains every architecture and training regime. Empirical and theoretical studies have instead identified several overlapping, context-dependent effects.^{[2][3][4][5][25]}

Original internal-covariate-shift account

Ioffe and Szegedy described "internal covariate shift" as changes in a layer's input distribution while earlier layers are updated. They proposed that standardizing intermediate inputs would make later layers easier to optimize, permit higher learning rates, reduce sensitivity to initialization, and sometimes reduce the need for dropout.^[1] Their experiments established the empirical training benefit for the models tested, but later studies challenged the proposed causal explanation.

Santurkar and colleagues deliberately injected time-varying mean and variance changes after BN layers. The perturbed networks retained optimization advantages even though the induced distribution shifts were large. They also found that BN made loss and gradient behavior smoother along measured optimization directions, which made gradients more predictive for larger steps in their experiments.^[2] These findings weaken a simple account in which stable activation distributions alone cause BN's success.

Learning rates, activation scale, and gradients

Bjorck and colleagues found that BN enabled larger learning rates in their experiments. In unnormalized deep networks, some large updates caused activation magnitudes to grow with depth and the loss to diverge. Repeated channel-wise correction of activation scale prevented that failure mode in the networks they studied.^[3] Their account connects faster convergence and some generalization gains to the learning-rate regime rather than to internal covariate shift by itself.

BN also makes a normalized linear transform largely insensitive to rescaling its incoming weights. Ignoring epsilon, (\mathrm{BN}(aWx)=\mathrm{BN}(Wx)) for positive scalar (a), while the gradient with respect to the rescaled weights changes inversely with (a). This scale invariance changes the geometry and effective step sizes of optimization.^[1][3]

In residual networks, De and Smith identified another architecture-specific effect. At initialization, BN reduces the relative scale of residual branches by a factor related to network depth, biasing deep residual blocks toward identity functions on average. They showed that this helps maintain well-behaved gradients in deep ResNets and used the observation to design an initialization for networks without normalization.^[4] This does not imply that BN is required for residual learning. Normalizer-Free ResNets later achieved high ImageNet accuracy using signal-propagation design and adaptive gradient clipping instead of normalization layers.^[19]

Stochastic coupling and regularization

Mini-batch statistics introduce stochasticity because each example is normalized using other randomly selected examples. The strength and character of that noise depend on the normalization batch, batch composition, spatial dimensions, and network. Ioffe and Szegedy observed that BN sometimes allowed them to remove dropout.^[1] Later work analyzed BN as an implicit regularizer and found interactions among its scale parameters, learning rate, and weight decay.^[21] These results support a regularization role, but they do not justify treating BN as a drop-in replacement for every explicit regularizer.

The coupling can also leak information between examples. If batches have an artificial structure, a model may exploit batch membership rather than learn a representation that works for independent predictions. Shuffling or synchronizing normalization batches can mitigate this in some multi-device and contrastive-learning settings.^[5]

Batch size and composition

Small normalization batches

BN needs enough representative values per feature to estimate useful moments. Small batches produce noisier estimates, and the gap between training-time batch statistics and inference statistics may grow. In the Group Normalization study, a ResNet-50 trained on ImageNet with two images per worker had 34.7% top-1 validation error with BN and 24.1% with Group Normalization, a 10.6 percentage-point difference under that study's training setup.^[7] The result is specific to the tested architecture and recipe, not a universal threshold at which BN fails.

An image batch size of one does not always imply zero variance. In spatial BN, one (H\times W) feature map still supplies (H W) values per channel. A dense BN feature with only one value in its reduction set does have zero empirical variance, but convolutional BN can have a larger reduction set even when (N=1).^[1][14] Correlation among nearby spatial values and poor representation of the data distribution can still make such statistics unsuitable.

The normalization batch is also different from the optimizer batch. Gradient accumulation combines gradients from several forward and backward passes, but ordinary BN computes statistics separately within each forward pass. Accumulation therefore increases the effective optimizer batch without increasing the BN normalization batch.^[5]

Distributed training

With data parallelism, ordinary BN typically computes statistics within each worker or device. The global optimizer batch may be large while every BN layer sees only the local examples. Synchronized batch normalization computes sums and squared sums across a process group so that workers use common statistics over a larger reduction set.^[5] PyTorch's SyncBatchNorm documentation specifies synchronization within a process group and supports conversion of existing BN modules for distributed data parallel training.^[15]

Synchronization adds collective communication at BN layers. It may help when per-device batches are too small, but it is not automatically superior. Local statistics can provide useful noise, and batches drawn from distinct domains should not necessarily be merged. The appropriate process group and normalization batch must match the data semantics as well as the hardware layout.^[5]

Non-independent and mixed-domain batches

Standard BN assumes that the values grouped for normalization provide a useful estimate for the population that will be represented by stored statistics. Class-homogeneous, temporally correlated, or otherwise non-independent batches can violate that assumption. Batch Renormalization was developed for small or non-independent mini-batches and corrects the current batch-normalized values toward the transform based on running statistics, with clipped correction terms during training.^[6]

When training and deployment domains differ, stored means and variances can encode the source domain. Adaptive Batch Normalization replaces source-domain statistics with moments estimated from target-domain data; the original AdaBN paper reported improvements on its domain-adaptation tasks without adding learned parameters.^[22] Schneider and colleagues similarly evaluated adapting BN statistics on unlabeled corrupted test data and found improved robustness in their experiments.^[23] Such adaptation is not universally safe: it assumes access to a representative target batch, can mix label or domain proportions in undesirable ways, and changes predictions according to other target samples.^[5]

Variants and alternatives

Scroll sideways for more →

Ghost BatchNorm deliberately divides a large batch into smaller normalization groups. Hoffer, Hubara, and Soudry introduced it while studying large-batch training and reported a reduced generalization gap in their experiments.^[12] It changes BN's noise without changing the number of examples contributing to the optimizer update.

PreciseBN changes how evaluation statistics are estimated rather than how the model is optimized. It runs data through a fixed model and aggregates moments, avoiding an exponential average over changing model states. Wu and Johnson reported stable estimates with roughly (10^3) to (10^4) images in their ImageNet experiments, but the required sample count depends on the model and distribution.^[5]

Layer normalization computes moments from features within one example, so its output does not depend on other batch members and it uses the same form during training and inference.^[8] Vanilla BN has performed poorly in standard transformer language models; the PowerNorm study attributed part of that result to large fluctuations in batch-dimension statistics for the tested NLP data and proposed a different running quadratic normalization.^[18] This evidence explains why substituting BN for LayerNorm is not a neutral implementation change, but it does not show that batch-derived normalization is impossible in all sequence models.

Instance Normalization computes per-example, per-channel spatial statistics and was introduced for fast image stylization.^[9] Group Normalization divides channels into groups and uses per-example statistics across each group and the spatial dimensions.^[7] RMSNorm is a LayerNorm variant that uses a root mean square without subtracting a feature mean; its authors reported similar task quality to LayerNorm with model- and implementation-dependent runtime reductions of 7% to 64% in their experiments.^[10]

Normalization-free designs are another alternative. Normalization Propagation analytically propagated activation moments under modeling assumptions instead of measuring each batch.^[24] NFNets combined scaled residual branches with adaptive gradient clipping and demonstrated that competitive large-scale image classifiers can be trained without BN.^[19] These methods replace BN with other architectural and optimization constraints rather than proving that normalization never helps.

Interactions with other components

Dropout

BN and dropout can be combined, but their order matters. Li and colleagues analyzed a variance shift when dropout changes the distribution feeding a later BN layer during training but is disabled at evaluation. Their experiments found that the effect depended on architecture, feature dimension, dropout form, and placement.^[11] A blanket rule that the two methods are incompatible would overstate the evidence. A safer design is to avoid placing dropout immediately before a BN whose stored statistics will be used at evaluation, unless that arrangement has been validated for the architecture.

Weight decay and affine parameters

Because normalization makes preceding weight scale partly redundant, weight decay can alter effective learning rates as well as directly regularize a function. Decaying BN's (\gamma) and (\beta) is not equivalent to decaying convolutional weights, and studies have reported task-dependent effects.^[13][21] Optimizer parameter groups should therefore follow a validated training recipe rather than assuming every parameter should receive identical decay.

Fine-tuning

Fine-tuning presents two separate choices: whether to update (\gamma,\beta), and whether to update the stored moments. Freezing affine parameters does not necessarily select evaluation statistics in every framework, and selecting evaluation mode can freeze other stateful layers as well. If the target data differ from pretraining data, old moments may be mismatched; if the target set or batch is small, newly estimated moments may be worse. Frozen BN, ordinary BN, and a post-training recalibration pass are distinct strategies that should be evaluated separately.^[5]

Framework behavior

Scroll sideways for more →

The table describes the cited versions, not an interchange guarantee.^[14][16] Exported models should be checked against the target operator semantics. ONNX version 15 specifies channel-wise reduction across every non-channel axis, float accumulation for its mean and variance with float16 input, and separate training and inference output forms.^[17]

A minimal PyTorch convolutional block is:

import torch.nn as nn

block = nn.Sequential(
    nn.Conv2d(3, 64, kernel_size=3, padding=1, bias=False),
    nn.BatchNorm2d(64),
    nn.ReLU(inplace=True),
)

block.train()  # Current-batch statistics; running estimates update
block.eval()   # Stored running statistics; no running-stat update

eval() does not disable gradient recording by itself, and requires_grad=False on affine parameters does not by itself switch the module to evaluation statistics. These are separate controls in PyTorch.^[14]

A corresponding TensorFlow block can pass the call mode explicitly:

import tensorflow as tf

conv = tf.keras.layers.Conv2D(64, 3, padding="same", use_bias=False)
bn = tf.keras.layers.BatchNormalization()
relu = tf.keras.layers.ReLU()

x = conv(inputs)
x = bn(x, training=training)
x = relu(x)

TensorFlow updates moving statistics when the BN layer is called in training mode. In the cited API, setting the BN layer's trainable property to false also makes that layer use moving statistics, a special behavior that differs from most Keras layers.^[16]

Diagnostic checklist

When a BN model behaves differently across training, validation, export, or deployment, the following checks isolate common causes:

Scroll sideways for more →

BN works reliably in many supervised CNN training pipelines with randomly sampled batches, sufficient reduction sets, and deployment data similar to training data. Outside that setting, the exact normalization batch and the treatment of stored statistics are part of the model definition, not incidental loader settings.^[5]

Historical significance

The original paper's combination of a differentiable mini-batch transform, learned affine recovery, and deterministic inference made activation normalization practical in large neural networks.^[1] ResNet then demonstrated BN throughout a deep residual image classifier.^[20] Research that followed separated several effects that the first explanation had grouped together: smoother measured optimization behavior, access to larger learning rates, architecture-specific signal scaling, stochastic regularization, and train-evaluation mismatch.^{[2][3][4][21]}

BN also motivated a family of normalization methods that choose different reduction dimensions or eliminate batch dependence, including LayerNorm, InstanceNorm, GroupNorm, and RMSNorm.^{[7][8][9][10]} Its lasting technical contribution is therefore broader than a claim that every network should normalize by mini-batch. It established normalization statistics, affine recovery, reduction axes, and training-inference state as explicit architectural choices.

References

1. ^Ioffe, S. and Szegedy, C. (2015). "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift." Proceedings of the 32nd International Conference on Machine Learning, PMLR 37:448-456. proceedings.mlr.press/...ioffe15.pdf
2. ^Santurkar, S., Tsipras, D., Ilyas, A., and Madry, A. (2018). "How Does Batch Normalization Help Optimization?" Advances in Neural Information Processing Systems 31. proceedings.neurips.cc/...60467e0a99e1cf-Paper.pdf
3. ^Bjorck, N., Gomes, C. P., Selman, B., and Weinberger, K. Q. (2018). "Understanding Batch Normalization." Advances in Neural Information Processing Systems 31. proceedings.neurips.cc/...5d704feb489480-Paper.pdf
4. ^De, S. and Smith, S. L. (2020). "Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks." Advances in Neural Information Processing Systems 33. proceedings.neurips.cc/...a9cbcba6c1881d-Paper.pdf
5. ^Wu, Y. and Johnson, J. (2021). "Rethinking 'Batch' in BatchNorm." arXiv:2105.07576. arxiv.org/...2105.07576
6. ^Ioffe, S. (2017). "Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models." Advances in Neural Information Processing Systems 30. papers.neurips.cc/...ced286cb5995327d1ab-Paper.pdf
7. ^Wu, Y. and He, K. (2018). "Group Normalization." Proceedings of the European Conference on Computer Vision, pp. 3-19. openaccess.thecvf.com/...ation_ECCV_2018_paper.pdf
8. ^Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). "Layer Normalization." arXiv:1607.06450. arxiv.org/...1607.06450
9. ^Ulyanov, D., Vedaldi, A., and Lempitsky, V. (2016). "Instance Normalization: The Missing Ingredient for Fast Stylization." arXiv:1607.08022. arxiv.org/...1607.08022
10. ^Zhang, B. and Sennrich, R. (2019). "Root Mean Square Layer Normalization." Advances in Neural Information Processing Systems 32. arxiv.org/...1910.07467
11. ^Li, X., Chen, S., Hu, X., and Yang, J. (2019). "Understanding the Disharmony Between Dropout and Batch Normalization by Variance Shift." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2682-2690. openaccess.thecvf.com/...iance_CVPR_2019_paper.pdf
12. ^Hoffer, E., Hubara, I., and Soudry, D. (2017). "Train Longer, Generalize Better: Closing the Generalization Gap in Large Batch Training of Neural Networks." Advances in Neural Information Processing Systems 30. proceedings.neurips.cc/...c7f1e88812af3d-Paper.pdf
13. ^Summers, C. and Dinneen, M. J. (2020). "Four Things Everyone Should Know to Improve Batch Normalization." International Conference on Learning Representations. arxiv.org/...1906.03548
14. ^PyTorch. "BatchNorm2d." PyTorch 2.9 documentation. docs.pytorch.org/....modules.batchnorm.BatchNorm2d
15. ^PyTorch. "SyncBatchNorm." PyTorch 2.9 documentation. docs.pytorch.org/...torch.nn.SyncBatchNorm
16. ^TensorFlow. "tf.keras.layers.BatchNormalization." TensorFlow 2.16.1 API documentation, updated June 7, 2024. tensorflow.org/...BatchNormalization
17. ^ONNX. "BatchNormalization, operator version 15." ONNX operator documentation. onnx.ai/...onnx__BatchNormalization
18. ^Shen, S., Yao, Z., Gholami, A., Mahoney, M. W., and Keutzer, K. (2020). "PowerNorm: Rethinking Batch Normalization in Transformers." Proceedings of the 37th International Conference on Machine Learning, PMLR 119:8741-8751. proceedings.mlr.press/...shen20e.pdf
19. ^Brock, A., De, S., Smith, S. L., and Simonyan, K. (2021). "High-Performance Large-Scale Image Recognition Without Normalization." Proceedings of the 38th International Conference on Machine Learning, PMLR 139:1059-1071. proceedings.mlr.press/...brock21a.pdf
20. ^He, K., Zhang, X., Ren, S., and Sun, J. (2016). "Deep Residual Learning for Image Recognition." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-778. openaccess.thecvf.com/...rning_CVPR_2016_paper.pdf
21. ^Luo, P., Wang, X., Shao, W., and Peng, Z. (2019). "Towards Understanding Regularization in Batch Normalization." International Conference on Learning Representations. arxiv.org/...1809.00846
22. ^Li, Y., Wang, N., Shi, J., Hou, X., and Liu, J. (2018). "Adaptive Batch Normalization for Practical Domain Adaptation." Pattern Recognition 80:109-117. arxiv.org/...1603.04779
23. ^Schneider, S., Rusak, E., Eck, L., Bringmann, O., Brendel, W., and Bethge, M. (2020). "Improving Robustness Against Common Corruptions by Covariate Shift Adaptation." Advances in Neural Information Processing Systems 33. proceedings.neurips.cc/...5c187784afc9ee-Paper.pdf
24. ^Arpit, D., Zhou, Y., Kota, B., and Govindaraju, V. (2016). "Normalization Propagation: A Parametric Technique for Removing Internal Covariate Shift in Deep Networks." Proceedings of the 33rd International Conference on Machine Learning, PMLR 48:1168-1176. proceedings.mlr.press/...arpitb16.pdf
25. ^Huang, L., Qin, J., Zhou, Y., Zhu, F., Liu, L., and Shao, L. (2023). "Normalization Techniques in Training DNNs: Methodology, Analysis and Application." IEEE Transactions on Pattern Analysis and Machine Intelligence 45(8):10173-10196. pubmed.ncbi.nlm.nih.gov/37027763