# Group normalization

> Source: https://aiwiki.ai/wiki/group_normalization
> Updated: 2026-11-18
> Fact-checked: 2026-11-18
> Categories: Computer Vision, Deep Learning, Neural Networks, Training & Optimization
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Group normalization." aiwiki.ai, 18 Nov 2026. https://aiwiki.ai/wiki/group_normalization
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Group normalization** (GN, also written GroupNorm) is a normalization layer for [deep neural networks](https://aiwiki.ai/wiki/deep_learning) that splits the channels of a feature map into groups and normalizes the features in each group, for each training example separately, using that group's own mean and variance. It was introduced by Yuxin Wu and [Kaiming He](https://aiwiki.ai/wiki/kaiming_he) of [Facebook AI Research](https://aiwiki.ai/wiki/meta_fair) in a paper first posted to arXiv on 22 March 2018 and presented at the European Conference on Computer Vision ([ECCV](https://aiwiki.ai/wiki/eccv)) 2018 [1][2]. Because its statistics never mix examples from the same mini-batch, group normalization behaves the same way at any [batch size](https://aiwiki.ai/wiki/batch_size), which made it a practical replacement for [batch normalization](https://aiwiki.ai/wiki/batch_normalization) in memory-hungry vision tasks such as [object detection](https://aiwiki.ai/wiki/object_detection), [instance segmentation](https://aiwiki.ai/wiki/instance_segmentation) and video classification, where memory limits keep the per-GPU batch small (detectors are commonly trained with one or two images per GPU) [1].

In the original experiments, a [ResNet](https://aiwiki.ai/wiki/resnet)-50 trained on [ImageNet](https://aiwiki.ai/wiki/imagenet) with two images per GPU reached 24.1% top-1 validation error with group normalization against 34.7% with batch normalization, a gap of 10.6 percentage points, while at the usual 32 images per GPU the two methods were within 0.5 points of each other [1]. Group normalization contains [layer normalization](https://aiwiki.ai/wiki/layer_normalization) and instance normalization as its two extreme cases [1]. It was later adopted in other model families, including the U-Net backbones of [denoising diffusion probabilistic models](https://aiwiki.ai/wiki/ddpm) and [Stable Diffusion](https://aiwiki.ai/wiki/stable_diffusion) [3][4].

## Background

Batch normalization, introduced by Ioffe and Szegedy in 2015, normalizes each channel using the mean and variance computed over the examples in a mini-batch [1]. Wu and He argued that this dependence on the batch dimension becomes a problem when the batch is small: the batch statistics are estimated inaccurately, and the model's error rises quickly as the batch shrinks [1]. In their setting, as in many libraries, "batch size" meant the number of examples per worker (GPU), because batch normalization statistics were computed on each worker and not shared across workers [1].

The constraint was most severe in [computer vision](https://aiwiki.ai/wiki/computer_vision) tasks that use high-resolution inputs. The paper noted that the Fast R-CNN, Faster R-CNN and [Mask R-CNN](https://aiwiki.ai/wiki/mask_r_cnn) frameworks trained with one or two images per GPU, and in that regime batch normalization was usually "frozen": its mean and variance were fixed from the pre-trained model and the layer was turned into a fixed linear transform [1]. In video classification with 3D convolutions, adding frames to each clip forced a smaller batch, so designers had to trade temporal length against batch size [1]. The authors also pointed out that batch normalization behaves differently at training time (batch statistics) and at test time (pre-computed running statistics), which creates inconsistencies when a model is transferred to a new task or data distribution [1].

Earlier alternatives already avoided the batch dimension. Layer normalization computes statistics over all channels of one example, instance normalization computes them over the spatial positions of one channel of one example, and weight normalization normalizes filter weights instead of features [1]. According to Wu and He, these methods worked well for recurrent and generative models but had not matched batch normalization's accuracy in visual recognition [1]. Other approaches kept batch statistics but tried to repair them: batch renormalization constrains the drift of the batch mean and variance, and synchronized batch normalization computes statistics across many GPUs, which the paper described as moving the problem to engineering and hardware rather than solving it [1].

<figure class="embedvideo">
<iframe src="https://www.youtube-nocookie.com/embed/m3TN9FFmqsI" title="Group Normalization"></iframe>
<figcaption>"Group Normalization", video from European Computer Vision Association, published December 4, 2018. <a href="https://www.youtube.com/watch?v=m3TN9FFmqsI">Watch on YouTube</a>.</figcaption>
</figure>

## Method

### Formulation

Wu and He described batch, layer, instance and group normalization as members of one family that differ only in which set of features shares a mean and standard deviation [1]. For a 2D image feature map indexed in $(N, C, H, W)$ order (batch, channel, height, width), each feature $x_i$ is normalized as

$$\hat{x}_i = \frac{1}{\sigma_i}\left(x_i - \mu_i\right)$$

with

$$\mu_i = \frac{1}{m}\sum_{k \in S_i} x_k, \qquad \sigma_i = \sqrt{\frac{1}{m}\sum_{k \in S_i}\left(x_k - \mu_i\right)^2 + \epsilon}$$

where $S_i$ is the set of features used to compute the statistics for $x_i$, $m$ is the size of that set, and $\epsilon$ is a small constant [1]. Every method in the family then applies a learned per-channel scale $\gamma$ and shift $\beta$, $y_i = \gamma \hat{x}_i + \beta$, to restore representational capacity lost by normalizing [1].

For group normalization the set is

$$S_i = \left\{ k \;\middle|\; k_N = i_N,\ \left\lfloor \frac{k_C}{C/G} \right\rfloor = \left\lfloor \frac{i_C}{C/G} \right\rfloor \right\}$$

where $G$ is the number of groups, $C/G$ is the number of channels per group, and $\lfloor \cdot \rfloor$ is the floor function [1]. In words, two features share statistics when they belong to the same example and to the same contiguous block of $C/G$ channels; the mean and standard deviation are taken over the spatial axes and over the channels of that group [1]. Groups are formed from consecutive channels along the channel axis [1]. $G$ is a [hyperparameter](https://aiwiki.ai/wiki/hyperparameter), and the paper used $G = 32$ by default [1].

| Method | Features that share one mean and variance | Uses other examples in the batch |
|---|---|---|
| Batch normalization | Same channel, across all examples and spatial positions | Yes |
| Layer normalization | Same example, across all channels and spatial positions | No |
| Instance normalization | Same example and same channel, across spatial positions | No |
| Group normalization | Same example and same channel group, across spatial positions | No |

Source: Wu and He's formulation [1].

### Relation to layer and instance normalization

Setting $G = 1$ puts every channel in one group, and group normalization becomes layer normalization; setting $G = C$ gives each channel its own group, and it becomes instance normalization [1]. The authors argued that layer normalization's assumption that all channels in a layer make similar contributions is less valid in convolutional networks, so letting each group have its own distribution gives group normalization more representational power; in their ImageNet runs it had lower training and validation error than layer normalization [1]. Instance normalization, at the other extreme, uses only spatial positions and cannot exploit dependence between channels [1].

### Motivation

The paper motivated grouping channels by analogy with hand-designed vision features. Descriptors such as SIFT, HOG and GIST are built from groups of histogram bins and are normalized group by group, and higher-level encodings such as VLAD and Fisher vectors are also group-wise [1]. Wu and He suggested that channels in a deep network are similarly not independent: a first-layer filter and its horizontal mirror image, for example, could be expected to produce similarly distributed responses on natural images and so could be normalized together [1]. They also cited a computational model from neuroscience in which responses are normalized across many cells, a computation described as occurring throughout the visual system [1].

### Implementation

The paper showed that group normalization takes a few lines in frameworks with automatic differentiation: reshape the $(N, C, H, W)$ tensor to $(N, G, C/G, H, W)$, compute the mean and variance over the last three axes, normalize, reshape back, and apply the per-channel $\gamma$ and $\beta$ [1]. Because the statistics come from the input example alone, the same computation runs at training and inference time, with no running averages to store [1][5].

[PyTorch](https://aiwiki.ai/wiki/pytorch) provides the layer as `torch.nn.GroupNorm(num_groups, num_channels, eps=1e-05, affine=True, device=None, dtype=None, *, bias=True)`. Its documentation states that the number of channels must be divisible by the number of groups, that the mean and standard deviation are calculated separately over each group, that the variance uses the biased estimator, and that the layer uses statistics computed from the input data in both training and evaluation modes [5]. Its examples note that `GroupNorm(6, 6)` is equivalent to instance normalization and `GroupNorm(1, 6)` to layer normalization for a six-channel input [5]. [Keras](https://aiwiki.ai/wiki/keras) offers `keras.layers.GroupNormalization`, with 32 groups and an epsilon of 0.001 by default; its documentation describes the single-group case as nearly identical to layer normalization [6].

## Experimental results

### ImageNet classification

Wu and He trained ResNet models on the roughly 1.28 million ImageNet training images on 8 GPUs, replacing batch normalization with each alternative while keeping the other hyperparameters fixed [1]. With 32 images per GPU, where batch normalization works well, the validation errors of ResNet-50 were [1]:

| Normalization | Top-1 validation error (%) | Difference from batch norm (points) |
|---|---|---|
| Batch normalization | 23.6 | 0 |
| Layer normalization | 25.3 | 1.7 |
| Instance normalization | 28.4 | 4.8 |
| Group normalization (G = 32) | 24.1 | 0.5 |

Group normalization had lower training error than batch normalization in this setting but slightly higher validation error. The authors attributed the gap to the regularizing effect of batch normalization's noisy batch statistics, which group normalization lacks, and suggested that pairing it with a suitable regularizer might close the gap [1]. A ResNet-50 trained with weight normalization reached 28.2% error [1].

The main result came from reducing the per-GPU batch while scaling the learning rate linearly with it (0.1 at 32 images, 0.1N/32 at N images) and keeping the number of epochs fixed [1]:

| Images per GPU | 32 | 16 | 8 | 4 | 2 |
|---|---|---|---|---|---|
| Batch norm error (%) | 23.6 | 23.7 | 24.8 | 27.3 | 34.7 |
| Group norm error (%) | 24.1 | 24.2 | 24.0 | 24.2 | 24.1 |

Batch normalization degraded sharply below 8 images per GPU, while group normalization stayed within a few tenths of a point across the whole range [1]. At 4 images per GPU, batch renormalization with tuned limits reached 26.3% error, better than batch normalization's 27.3% but 2.1 points worse than group normalization [1]. On the deeper ResNet-101, group normalization was 0.4 points worse than batch normalization at 32 images per GPU (22.4% against 22.0%) and 8.9 points better at 2 images per GPU (23.0% against 31.9%) [1]. On VGG-16, which can be trained without any normalization, group normalization reached 27.6% error, against 28.0% for batch normalization and 29.2% without normalization [1].

### Number of groups

With a fixed number of groups, ResNet-50 errors ranged from 24.1% (G = 32) to 24.7% (G = 2), while the single-group case, equivalent to layer normalization, reached 25.3% [1]. With a fixed number of channels per group, errors ranged from 24.2% (16 channels per group) to 25.6% (2 channels per group), and one channel per group, equivalent to instance normalization, gave 28.4% [1]. The authors read the result as showing that grouping channels itself matters: even two channels per group was clearly better than instance normalization [1].

### Object detection and segmentation on COCO

The authors fine-tuned ImageNet-pretrained models for Mask R-CNN on the [COCO dataset](https://aiwiki.ai/wiki/coco_dataset), using the Detectron codebase, one image per GPU on 8 GPUs, and no weight decay on $\gamma$ and $\beta$ [1]. The baseline used frozen batch normalization. With a ResNet-50 C4 backbone, group normalization improved box AP by 1.1 and mask AP by 0.8, even though the GN-pretrained ImageNet model was slightly less accurate than the BN one [1]. A layer-normalization variant was 1.9 box AP worse than group normalization [1].

With a [Feature Pyramid Network](https://aiwiki.ai/wiki/feature_pyramid_network) backbone, adding group normalization to the convolutional box head alone raised box AP from 38.6 to 39.5, and using the GN-pretrained backbone as well raised it to 40.0 [1]. Applying batch normalization to the box head instead was about 9 AP worse, which the authors linked to the regions of interest in a batch all coming from the same image and therefore not being independent samples [1]. With GN in the backbone, box head and mask head, a ResNet-50 model trained on a longer schedule reached 40.8 box AP and 36.1 mask AP, 2.2 and 1.6 points above the frozen-BN baseline; the ResNet-101 counterpart reached 42.3 and 37.2 [1].

Group normalization also made it straightforward to train Mask R-CNN from random initialization. Trained for 270,000 iterations without ImageNet pre-training, a ResNet-101 model reached 41.0 box AP and 36.4 mask AP, which the authors described as the best from-scratch COCO results reported up to that time [1]. The code and models were released in the Detectron repository, including ImageNet-pretrained ResNet-50 and ResNet-101 models trained with 32 groups [7].

### Video classification on Kinetics

On the Kinetics action-recognition dataset, the authors used a ResNet-50 Inflated 3D (I3D) network and extended the normalization to cover the temporal axis as well as the spatial ones [1]. With 32-frame clips, group normalization was 0.3 points behind batch normalization in top-1 accuracy at 8 clips per GPU (73.0% against 73.3%) but ahead at 4 clips per GPU (72.8% against 72.1%) [1]. With 64-frame clips, which only fit at 4 clips per GPU, group normalization reached 74.5% top-1 accuracy against 73.3% for batch normalization; the authors argued that batch normalization's small-batch penalty had been hiding the benefit of longer clips [1].

## Publication and reception

The arXiv preprint (arXiv:1803.08494) went through three versions; the third, posted in June 2018, updated the from-scratch COCO result to 41.0 AP [2]. The paper appeared in the ECCV 2018 proceedings (Lecture Notes in Computer Science, pages 3 to 19), was presented as an oral, and received a Best Paper Honorable Mention at the conference [8][9]. A journal version was published online in the International Journal of Computer Vision in July 2019 and appeared in volume 128, issue 3 (March 2020), pages 742 to 755 [10]. As of November 2026, the OpenAlex scholarly index, which merges the arXiv, ECCV and IJCV versions into one record, counted about 5,200 citations of the paper [11].

## Adoption

### Detection and training from scratch

Group normalization was one of the two normalization schemes, with synchronized batch normalization, that He, Ross Girshick and Piotr Dollár used in "Rethinking ImageNet Pre-training", which studied training COCO detectors from random initialization [12]. They reported that both GN and synchronized BN allowed detection models to be trained from scratch, and their main learning-curve comparisons used Mask R-CNN with FPN backbones and group normalization [12].

### Weight standardization and Big Transfer

Siyuan Qiao and colleagues proposed weight standardization in 2019, which standardizes convolution weights to zero mean and unit variance, and reported that combining it with group normalization narrowed the gap to batch normalization: on ImageNet, GN plus weight standardization trained with one image per GPU matched or outperformed batch normalization trained with large batches, and on COCO it significantly improved micro-batch training [13]. The Big Transfer (BiT) models of Alexander Kolesnikov and colleagues replaced every batch normalization layer in their ResNet-v2 architecture with group normalization and used weight standardization in all convolutional layers [14]. The BiT authors reported that GN alone did not scale to very large batches (a 5.4-point drop in ImageNet top-1 accuracy against batch normalization for a ResNet-50 trained with a batch of 4,096 on 128 accelerator chips), that adding weight standardization fixed this, and that the GN and weight standardization combination transferred better than batch normalization [14].

### Diffusion models

Jonathan Ho, Ajay Jain and Pieter Abbeel's 2020 [denoising diffusion probabilistic model](https://aiwiki.ai/wiki/ddpm) used a U-Net backbone "with group normalization throughout", replacing the weight normalization of the PixelCNN++ architecture it was based on to simplify the implementation [3]. Stable Diffusion is a latent diffusion model, and the CompVis Stable Diffusion repository defines its standard normalization layer as a GroupNorm with 32 groups, which the blocks of its U-Net call throughout [4][18]. The CompVis latent-diffusion autoencoder code likewise uses GroupNorm with 32 groups by default [15]. AI Wiki's [U-Net](https://aiwiki.ai/wiki/unet) article describes how these layers fit into the diffusion U-Net.

## Limitations and later work

The original paper acknowledged that group normalization was slightly less accurate than batch normalization when batches were large, which it attributed to the missing regularization from batch noise, and that many state-of-the-art systems and their hyperparameters had been designed around batch normalization, so they might not be optimal for GN-based models [1].

Later papers framed themselves partly as responses to these limits. Saurabh Singh and Shankar Krishnan's Filter Response Normalization (2019) argued that alternatives such as batch renormalization and group normalization either do not match batch normalization at large batches, still degrade at smaller batches, or constrain the architecture, and reported that their method was more than 1% better than GN on ImageNet in the small-batch regime [16]. In both the weight standardization experiments, with one image per GPU, and the BiT experiments, with a batch of 4,096, group normalization performed better when combined with weight standardization than on its own [13][14]. Other work moved away from normalization layers altogether: Andrew Brock and colleagues' Normalizer-Free ResNets (2021) were motivated by batch normalization's dependence on batch size and interactions between examples, and used adaptive gradient clipping instead of normalization layers [17].

## See also

- [Normalization](https://aiwiki.ai/wiki/normalization)
- [Batch normalization](https://aiwiki.ai/wiki/batch_normalization)
- [Layer normalization](https://aiwiki.ai/wiki/layer_normalization)
- [RMSNorm](https://aiwiki.ai/wiki/rmsnorm)
- [Convolutional neural network](https://aiwiki.ai/wiki/convolutional_neural_network)
- [Diffusion model](https://aiwiki.ai/wiki/diffusion_model)

## References

1. Wu, Yuxin and He, Kaiming. "Group Normalization". arXiv:1803.08494v3 (Facebook AI Research), June 2018. https://arxiv.org/abs/1803.08494
2. arXiv. "Group Normalization" (arXiv:1803.08494), abstract page and version history, submitted 2018-03-22. https://arxiv.org/abs/1803.08494v3
3. Ho, Jonathan; Jain, Ajay; Abbeel, Pieter. "Denoising Diffusion Probabilistic Models". arXiv:2006.11239, 2020. https://arxiv.org/abs/2006.11239
4. CompVis. "stable-diffusion: ldm/modules/diffusionmodules/util.py". GitHub. https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/diffusionmodules/util.py
5. PyTorch. "GroupNorm". PyTorch documentation, version 2.15. https://docs.pytorch.org/docs/stable/generated/torch.nn.GroupNorm.html
6. Keras. "GroupNormalization layer". Keras 3 API documentation. https://keras.io/api/layers/normalization_layers/group_normalization/
7. Facebook Research. "Group Normalization for Mask R-CNN". Detectron repository, GitHub. https://github.com/facebookresearch/Detectron/blob/main/projects/GN/README.md
8. Wu, Yuxin and He, Kaiming. "Group Normalization". Computer Vision ECCV 2018, Lecture Notes in Computer Science, pp. 3-19, Springer, 2018. https://doi.org/10.1007/978-3-030-01261-8_1
9. He, Kaiming. "Kaiming He" (publications and awards). MIT CSAIL personal homepage. https://people.csail.mit.edu/kaiming/
10. Wu, Yuxin and He, Kaiming. "Group Normalization". International Journal of Computer Vision 128(3), pp. 742-755, published online 2019-07-22. https://doi.org/10.1007/s11263-019-01198-w
11. OpenAlex. "Group Normalization" (work W2795783309, merged record with citation count), accessed November 2026. https://api.openalex.org/works/W2795783309
12. He, Kaiming; Girshick, Ross; Dollár, Piotr. "Rethinking ImageNet Pre-training". arXiv:1811.08883, 2018. https://arxiv.org/abs/1811.08883
13. Qiao, Siyuan; Wang, Huiyu; Liu, Chenxi; Shen, Wei; Yuille, Alan. "Micro-Batch Training with Batch-Channel Normalization and Weight Standardization". arXiv:1903.10520, 2019. https://arxiv.org/abs/1903.10520
14. Kolesnikov, Alexander; Beyer, Lucas; Zhai, Xiaohua; Puigcerver, Joan; Yung, Jessica; Gelly, Sylvain; Houlsby, Neil. "Big Transfer (BiT): General Visual Representation Learning". arXiv:1912.11370, 2019. https://arxiv.org/abs/1912.11370
15. CompVis. "latent-diffusion: ldm/modules/diffusionmodules/model.py". GitHub. https://github.com/CompVis/latent-diffusion/blob/main/ldm/modules/diffusionmodules/model.py
16. Singh, Saurabh and Krishnan, Shankar. "Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks". arXiv:1911.09737, 2019. https://arxiv.org/abs/1911.09737
17. Brock, Andrew; De, Soham; Smith, Samuel L.; Simonyan, Karen. "High-Performance Large-Scale Image Recognition Without Normalization". arXiv:2102.06171, 2021. https://arxiv.org/abs/2102.06171
18. CompVis. "stable-diffusion: ldm/modules/diffusionmodules/openaimodel.py". GitHub. https://github.com/CompVis/stable-diffusion/blob/main/ldm/modules/diffusionmodules/openaimodel.py

