# Dropout Regularization

> Source: https://aiwiki.ai/wiki/dropout_regularization
> Updated: 2026-06-21
> Categories: Deep Learning, Machine Learning, Training & Optimization
> From AI Wiki (https://aiwiki.ai), a free encyclopedia of artificial intelligence. Quote with attribution.

*See also: [Machine learning terms](/wiki/machine_learning_terms), [Regularization](/wiki/regularization)*

**Dropout regularization** is a [regularization](/wiki/regularization) technique for [neural networks](/wiki/neural_network) that prevents [overfitting](/wiki/overfitting) by randomly setting a fraction of neuron activations to zero on each training step, forcing the network to learn redundant, distributed representations instead of relying on co-adapted features. It was introduced by Geoffrey Hinton and collaborators in 2012 and formalized by Nitish Srivastava et al. in the 2014 Journal of Machine Learning Research paper "Dropout: A Simple Way to Prevent Neural Networks from Overfitting," which by 2026 had accumulated more than 43,000 citations and made dropout one of the most widely used regularization methods in deep learning.[1][2][13] As the 2014 paper states, "The key idea is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much."[2]

## Origins and history

The idea of dropout was first introduced in a 2012 paper by Hinton, Srivastava, Krizhevsky, Sutskever, and Salakhutdinov titled "Improving neural networks by preventing co-adaptation of feature detectors." The paper demonstrated that randomly omitting half of the feature detectors on each training case significantly reduced overfitting and produced state-of-the-art results on speech and object recognition benchmarks.[1] In the authors' words, this "overfitting is greatly reduced by randomly omitting half of the feature detectors on each training case," so that "each neuron learns to detect a feature that is generally helpful for producing the correct answer given the combinatorially large variety of internal contexts in which it must operate."[1]

The technique was then formalized and studied extensively in the 2014 paper "Dropout: A Simple Way to Prevent Neural Networks from Overfitting" by Srivastava, Hinton, Krizhevsky, Sutskever, and Salakhutdinov, published in the Journal of Machine Learning Research (JMLR), volume 15, pages 1929-1958.[2] This paper provided a thorough empirical evaluation across vision, speech recognition, document classification, and computational biology tasks, alongside theoretical analysis connecting dropout to model averaging and Bayesian inference. The paper frames the method as efficient model averaging: "During training, dropout samples from an exponential number of different thinned networks. At test time, it is easy to approximate the effect of averaging the predictions of all these thinned networks by simply using a single unthinned network that has smaller weights."[2]

Hinton has noted that the biological inspiration for dropout came from the observation that sexual reproduction, which randomly combines half the genes from each parent, is more effective at producing robust organisms than asexual reproduction. In a similar way, dropout forces each neuron to be useful on its own and in combination with random subsets of other neurons.[2]

## How does dropout work?

Dropout operates differently during training and inference. Understanding both phases is essential for grasping why the technique is effective.

### Training phase

During each forward pass in training, dropout randomly "drops" (sets to zero) each neuron's output with probability *p*, called the dropout rate. Only the surviving neurons participate in the forward pass and the subsequent backpropagation step. The dropped neurons receive no gradient updates for that particular training iteration.

Mathematically, for a layer with output vector **h**, dropout applies an element-wise mask:

```
r_j ~ Bernoulli(1 - p)
h'_j = r_j * h_j
```

where *r_j* is a binary random variable that equals 1 with probability (1 - p) and 0 with probability *p*. The vector **h'** replaces **h** as input to the next layer. Because the mask is resampled for every training example (or mini-batch), the network effectively trains a different "thinned" sub-network on each step.[2]

### Inference phase

At test time, all neurons are active (no dropout is applied). To compensate for the fact that more neurons are active during inference than during any single training step, the outputs must be scaled. In the original formulation, each neuron's output is multiplied by (1 - p) at test time so that the expected output matches what was seen during training.[2]

### Inverted dropout

Modern deep learning frameworks use **inverted dropout**, which handles the scaling during training rather than at inference. During training, surviving activations are multiplied by 1 / (1 - p), so that the expected value of the output remains unchanged regardless of dropout. At test time, the network is used as-is with no modifications. This approach is preferred because it simplifies the inference code and avoids any test-time overhead. Both PyTorch and TensorFlow/Keras implement inverted dropout by default.

The inverted dropout formulation is:

```
r_j ~ Bernoulli(1 - p)
h'_j = (r_j * h_j) / (1 - p)
```

During inference, h'_j = h_j (no change needed).

## How do you choose the dropout rate?

The dropout rate *p* is a hyperparameter that controls how aggressively neurons are deactivated. The optimal value depends on the network architecture, dataset size, and the degree of overfitting observed.

| Layer type | Typical dropout rate (*p*) | Notes |
|---|---|---|
| Input layer | 0.0 to 0.2 | Dropping too many input features can lose critical information |
| Hidden layers (fully connected) | 0.3 to 0.5 | 0.5 is the most common default; higher values for large networks |
| [Convolutional neural network](/wiki/convolutional_neural_network) layers | 0.1 to 0.3 | Convolutional layers have fewer parameters per feature map, so less dropout is needed |
| [Recurrent neural network](/wiki/recurrent_neural_network) layers | 0.2 to 0.5 | Applied carefully to avoid disrupting temporal dependencies |
| Output layer | 0.0 | Dropout is almost never applied to the output layer |

Srivastava et al. (2014) found that a retention probability of 0.5 for hidden units and 0.8 for input units worked well across a wide range of tasks (equivalently, dropout rates of 0.5 and 0.2).[2] A useful guideline is that if the network is underfitting, the dropout rate should be lowered (or dropout removed entirely), and if overfitting is severe, the rate can be increased or the network can be made wider to compensate.

## Why does dropout work?

Several complementary explanations have been proposed for why dropout is effective.

### Ensemble interpretation

A neural network with *n* units that can be dropped has 2^n possible thinned sub-networks. Training with dropout can be viewed as training an exponential number of weight-sharing sub-networks simultaneously. At test time, using the full network with scaled weights approximates the geometric mean of the predictions from all these sub-networks.[2] This is analogous to [ensemble](/wiki/ensemble) methods like bagging, where multiple models are trained on different subsets of data and their predictions are averaged. The key difference is that dropout achieves this effect within a single network without the computational cost of training separate models.

### Preventing co-adaptation

Without dropout, neurons can develop complex co-adaptations where a neuron becomes useful only in the presence of specific other neurons. This makes the network brittle because it depends on particular feature combinations. Dropout breaks these co-adaptations by ensuring that a neuron cannot rely on any specific set of partner neurons being present. Each neuron is forced to learn features that are individually useful across many random contexts, producing more robust and transferable representations.[1]

### Implicit regularization

Dropout adds noise to the training process, which acts as a form of regularization similar to adding noise to the weights or the gradients. Wager, Wang, and Liang (2013) showed that for generalized linear models, the regularization effect of dropout is approximately equivalent to an adaptive form of L2 regularization where the penalty depends on the Fisher information of each feature.[4] This provides a theoretical link between dropout and classical regularization techniques.

### Bayesian interpretation

Gal and Ghahramani (2016) established a formal connection between dropout and approximate Bayesian inference. They showed that a neural network with dropout applied before every weight layer is mathematically equivalent to an approximation of a deep Gaussian process.[7] This interpretation opened the door to using dropout for uncertainty estimation (see the Monte Carlo dropout section below).

## What are the main dropout variants?

Since the introduction of standard dropout, researchers have developed several variants tailored to different architectures and use cases.

### Spatial dropout

Standard element-wise dropout is not ideal for [convolutional neural network](/wiki/convolutional_neural_network) layers because adjacent pixels in a feature map are highly correlated. Dropping individual pixels has little effect since the information can be recovered from neighboring activations. Spatial dropout (Tompson et al., 2015) addresses this by dropping entire feature maps (channels) rather than individual elements.[5] If a feature map is dropped, all spatial locations within that map are set to zero simultaneously. This forces the network to learn to use information from multiple feature maps rather than relying on any single one.

### DropConnect

DropConnect (Wan et al., 2013) generalizes dropout by randomly setting individual weights (connections) to zero rather than setting entire neuron activations to zero.[3] While dropout masks the output of neurons, DropConnect masks the weight matrix itself. This means each neuron receives input from a random subset of the neurons in the previous layer. DropConnect is more fine-grained than dropout and was shown to improve performance on several image classification benchmarks, though it is computationally more expensive and less commonly used in practice.[3]

### DropBlock

DropBlock (Ghiasi, Lin, and Le, 2018) extends spatial dropout specifically for convolutional networks by dropping contiguous rectangular regions of a feature map.[11] Rather than dropping individual elements or entire channels, DropBlock removes a block of spatially correlated units. This is more effective for convolutional layers because it forces the network to look at a wider spatial context rather than relying on localized features. DropBlock was shown to outperform both standard dropout and spatial dropout on ImageNet and COCO benchmarks.[11]

### Variational dropout

Kingma, Salimans, and Welling (2015) proposed variational dropout, which frames dropout as a form of variational inference.[6] Unlike standard dropout where the same dropout rate is applied uniformly, variational dropout can learn individual dropout rates for each weight or unit. Gal and Ghahramani (2016) extended this idea by showing that applying the same dropout mask at each time step of a [recurrent neural network](/wiki/recurrent_neural_network) (rather than resampling at every step) corresponds to a form of approximate variational inference.[7] This approach, sometimes called "locked" or "tied" dropout, prevents the recurrent connections from losing long-term memory due to accumulating dropout noise over time.

### Alpha dropout

Alpha dropout (Klambauer et al., 2017) was designed specifically for self-normalizing neural networks that use the Scaled Exponential Linear Unit (SELU) activation function.[9] Standard dropout disrupts the mean and variance properties that SELU depends on for self-normalization. Alpha dropout instead sets dropped activations to the negative saturation value of SELU (rather than zero) and applies an affine transformation to maintain the desired mean and variance.[9] This preserves the self-normalizing property throughout the network.

### Concrete dropout

Concrete dropout (Gal, Hron, and Kendall, 2017) treats the dropout rate as a learnable parameter that is optimized jointly with the network weights during training.[8] It uses a continuous relaxation of the discrete Bernoulli distribution (the "concrete" distribution) to allow gradient-based optimization of the dropout probability.[8] This eliminates the need to manually tune dropout rates, which is particularly useful in [Bayesian neural network](/wiki/bayesian_neural_network) settings where the dropout rate has a principled interpretation as a prior parameter.

| Variant | What is dropped | Key advantage | Best suited for |
|---|---|---|---|
| Standard dropout | Individual activations | Simple, general-purpose | Fully connected layers |
| Spatial dropout | Entire feature maps (channels) | Respects spatial correlations | Convolutional layers |
| DropConnect | Individual weights | More fine-grained noise | Fully connected layers |
| DropBlock | Contiguous spatial regions | Forces wider spatial context | Convolutional layers |
| Variational dropout | Activations (with learned rates) | Consistent mask across time steps | Recurrent layers |
| Alpha dropout | Activations (to SELU saturation value) | Preserves self-normalization | SELU-based networks |
| Concrete dropout | Activations (learned rate) | No manual tuning of dropout rate | Bayesian settings |

## How is dropout used in transformers?

The [transformer](/wiki/transformer) architecture applies dropout in multiple locations. The original "Attention Is All You Need" paper (Vaswani et al., 2017) used dropout in three places: after the attention weight computation (attention dropout), after each sub-layer output before the residual addition, and within the feed-forward sub-layers.[10] The base transformer model used a residual dropout rate of P_drop = 0.1.[10] Typical dropout rates in transformers range from 0.1 to 0.3.

**Attention dropout** randomly zeros out entries in the attention weight matrix, which encourages the model to distribute attention across a broader set of positions rather than concentrating on a few tokens. **Residual dropout** is applied to the output of each sub-layer before it is added to the residual connection.[10] Some implementations also apply dropout to the embedding layer outputs.

Large-scale language models such as GPT-2 and BERT use dropout as one of their primary regularization mechanisms. Interestingly, some very large models (GPT-3, for example) have been trained without dropout, relying instead on the sheer volume of training data to prevent overfitting. Recent work by Liu et al. (2023) showed that dropout can also reduce underfitting in certain training regimes when applied with a scheduled rate.

## Monte Carlo dropout for uncertainty estimation

One of the most influential extensions of dropout is Monte Carlo (MC) dropout, proposed by Gal and Ghahramani (2016).[7] The key insight is that keeping dropout active at inference time and running multiple forward passes with different random masks produces a distribution of predictions rather than a single point estimate.[7]

The procedure is:

1. Train the network with dropout as usual.
2. At inference time, keep dropout enabled.
3. Run *T* forward passes on the same input, each with a different dropout mask.
4. The mean of the *T* predictions approximates the predictive mean.
5. The variance of the *T* predictions provides an estimate of model (epistemic) uncertainty.

Gal and Ghahramani proved that this procedure is mathematically equivalent to approximate inference in a deep Gaussian process, giving it a principled Bayesian interpretation.[7] MC dropout is attractive in practice because it requires no changes to the model architecture or training procedure; the only cost is running multiple forward passes at test time. It has been applied in safety-critical domains such as medical imaging, autonomous driving, and robotics, where knowing how confident a model is can be as important as the prediction itself.

## How does dropout interact with batch normalization?

[Batch normalization](/wiki/batch_normalization) (BN) and dropout are both widely used regularization techniques, but combining them can be problematic. Li et al. (2019) published a paper titled "Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift" that analyzed why the two techniques sometimes interfere.[12]

The core issue is that dropout introduces a variance shift between training and inference. During training, the random zeroing of activations changes the distribution of inputs to the batch normalization layer. BN computes running statistics (mean and variance) during training that it uses at test time, but these statistics are computed on dropout-perturbed activations. At inference time, dropout is turned off, so the actual activation distribution differs from what BN expects. This mismatch can degrade performance.[12]

Several strategies have been proposed to address this:

- **Place dropout after batch normalization:** applying dropout after BN (in the order: Linear/Conv, BN, Activation, Dropout) tends to be more stable than the reverse.
- **Use dropout only in the final fully connected layers:** many modern convolutional architectures (e.g., ResNet) use BN throughout but only apply dropout in the classifier head.
- **Skip dropout entirely when using BN:** batch normalization already provides a regularization effect through mini-batch noise, so additional dropout may be unnecessary.
- **Use Gaussian dropout instead of Bernoulli dropout:** Gaussian dropout applies continuous noise rather than binary masking, which reduces the variance shift problem.

## When should you not use dropout?

Dropout is not universally beneficial and can hurt performance in certain situations:

- **Small datasets with small models:** if the model is not overfitting, dropout may cause underfitting by reducing the effective capacity of the network.
- **When batch normalization provides sufficient regularization:** adding dropout on top of BN can create the variance shift problem described above without meaningful gains.
- **Very large training sets:** models trained on massive datasets (hundreds of millions of examples) may not overfit at all, making dropout unnecessary. GPT-3, for instance, was trained without dropout.
- **Tasks requiring precise output scaling:** in regression problems where exact magnitudes matter, the stochastic noise from dropout can interfere with learning precise output values.
- **Already sparse architectures:** networks with inherently sparse activations (e.g., networks using ReLU with many dead units) may not benefit from additional sparsity through dropout.

## Implementation

Dropout is straightforward to implement in modern deep learning frameworks.

### PyTorch

PyTorch provides several dropout modules:

```python
import torch.nn as nn

# Standard dropout (for fully connected layers)
model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Dropout(p=0.5),       # drops 50% of activations
    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Dropout(p=0.5),
    nn.Linear(128, 10)
)

# Dropout2d (for convolutional layers - drops entire channels)
conv_block = nn.Sequential(
    nn.Conv2d(64, 128, kernel_size=3, padding=1),
    nn.BatchNorm2d(128),
    nn.ReLU(),
    nn.Dropout2d(p=0.2)      # drops 20% of feature maps
)
```

PyTorch automatically disables dropout when `model.eval()` is called and re-enables it with `model.train()`. Additional variants include `nn.Dropout1d` for temporal data and `nn.Dropout3d` for volumetric data.

### TensorFlow / Keras

```python
import tensorflow as tf

model = tf.keras.Sequential([
    tf.keras.layers.Dense(256, activation='relu', input_shape=(784,)),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(128, activation='relu'),
    tf.keras.layers.Dropout(0.5),
    tf.keras.layers.Dense(10, activation='softmax')
])

# For convolutional layers (spatial dropout)
conv_model = tf.keras.Sequential([
    tf.keras.layers.Conv2D(64, (3, 3), activation='relu'),
    tf.keras.layers.SpatialDropout2D(0.2),
    tf.keras.layers.Conv2D(128, (3, 3), activation='relu'),
    tf.keras.layers.SpatialDropout2D(0.2)
])

# Alpha dropout for SELU networks
selu_model = tf.keras.Sequential([
    tf.keras.layers.Dense(256, activation='selu'),
    tf.keras.layers.AlphaDropout(0.1)
])
```

Keras automatically handles the training/inference distinction: dropout is active during `model.fit()` and `model.train_on_batch()` but inactive during `model.predict()` and `model.evaluate()`.

## Explain like I'm 5 (ELI5)

Imagine you are on a team of kids trying to solve a jigsaw puzzle. Every time you practice, some of your teammates have to sit out at random. Because you never know who will be missing, every kid on the team learns to be helpful on their own, not just when their best friend is there. When the real contest comes and everyone shows up, the whole team is much stronger because every member knows how to contribute.

Dropout works the same way in a neural network. During practice (training), random neurons are told to sit out. This forces the remaining neurons to learn useful features on their own. When the network takes a real test (inference), all neurons participate, and the network performs better because no single neuron is a weak link that the others depend on too much.

## References

1. Hinton, G. E., Srivastava, N., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. R. (2012). "Improving neural networks by preventing co-adaptation of feature detectors." arXiv preprint arXiv:1207.0580. https://arxiv.org/abs/1207.0580
2. Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). "Dropout: A Simple Way to Prevent Neural Networks from Overfitting." *Journal of Machine Learning Research*, 15(56), 1929-1958. https://jmlr.org/papers/v15/srivastava14a.html
3. Wan, L., Zeiler, M., Zhang, S., Le Cun, Y., & Fergus, R. (2013). "Regularization of Neural Networks using DropConnect." *Proceedings of the 30th International Conference on Machine Learning (ICML)*, 1058-1066.
4. Wager, S., Wang, S., & Liang, P. (2013). "Dropout Training as Adaptive Regularization." *Advances in Neural Information Processing Systems (NeurIPS)*, 26.
5. Tompson, J., Goroshin, R., Jain, A., LeCun, Y., & Breaux, C. (2015). "Efficient Object Localization Using Convolutional Networks." *Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)*.
6. Kingma, D. P., Salimans, T., & Welling, M. (2015). "Variational Dropout and the Local Reparameterization Trick." *Advances in Neural Information Processing Systems (NeurIPS)*, 28.
7. Gal, Y. & Ghahramani, Z. (2016). "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning." *Proceedings of the 33rd International Conference on Machine Learning (ICML)*, 1050-1059.
8. Gal, Y., Hron, J., & Kendall, A. (2017). "Concrete Dropout." *Advances in Neural Information Processing Systems (NeurIPS)*, 30.
9. Klambauer, G., Unterthiner, T., Mayr, A., & Hochreiter, S. (2017). "Self-Normalizing Neural Networks." *Advances in Neural Information Processing Systems (NeurIPS)*, 30.
10. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L., & Polosukhin, I. (2017). "Attention Is All You Need." *Advances in Neural Information Processing Systems (NeurIPS)*, 30.
11. Ghiasi, G., Lin, T.-Y., & Le, Q. V. (2018). "DropBlock: A regularization method for convolutional neural networks." *Advances in Neural Information Processing Systems (NeurIPS)*, 31.
12. Li, X., Chen, S., Hu, X., & Yang, J. (2019). "Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift." *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2682-2690.
13. Semantic Scholar. "Dropout: a simple way to prevent neural networks from overfitting" (paper record, citation count). Retrieved June 2026. https://www.semanticscholar.org/paper/34f25a8704614163c4095b3ee2fc969b60de4698