Dropout
Last reviewed
Sources
19 citations
Review status
Source-backed
Revision
v4 · 7,046 words
Improve this article
Add missing citations, update stale details, or suggest a clearer explanation.
Suggest edit
CtrlK
Last reviewed
Sources
19 citations
Review status
Source-backed
Revision
v4 · 7,046 words
Add missing citations, update stale details, or suggest a clearer explanation.
Dropout is a regularization technique for neural networks that randomly sets a fraction of neuron activations to zero during training, forcing the network to learn redundant representations that do not depend on any single neuron. It was proposed by Geoffrey Hinton and colleagues in a July 2012 arXiv preprint [1] and described in full by Nitish Srivastava, Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov in a 2014 paper in the Journal of Machine Learning Research (JMLR), volume 15, pages 1929-1958 [2]. As the JMLR paper states, "The key idea is to randomly drop units (along with their connections) from the neural network during training. This prevents units from co-adapting too much" [2]. At each training step, every neuron in a given layer has a probability p of being temporarily "dropped out" (its output set to zero); at test time the full network is used with rescaled weights, which approximates averaging an exponential number of thinned networks. Dropout became one of the most important tools for preventing overfitting in deep learning, and the 2014 JMLR paper has been cited more than 42,000 times according to Semantic Scholar [19], making it one of the most cited works in machine learning.
Dropout was a breakthrough when it was introduced. Before dropout, training deep neural networks on small to medium datasets almost always led to severe overfitting, where the model would memorize the training data but fail to generalize to new inputs. Weight decay, early stopping, and data augmentation helped somewhat, but dropout provided a qualitatively different kind of regularization that dramatically improved generalization across many tasks: image classification, speech recognition, document classification, and computational biology [2]. The original 2012 preprint reported that "Random 'dropout' gives big improvements on many benchmark tasks and sets new records for speech and object recognition" [1].
Dropout works like temporarily and randomly removing workers from a team during every practice session so that the team never becomes dependent on any one person. In a neural network, each "worker" is a neuron. During training, dropout randomly switches off a fraction of neurons on every forward pass, so the remaining neurons must learn to do useful work even when many of their usual collaborators are missing. Because a different random subset is removed each time, the network ends up training a huge collection of overlapping "thinned" sub-networks that share weights. At test time all neurons are switched back on, and the single full network behaves like an average of all those sub-networks. The result is a model that generalizes better to data it has not seen.
The idea behind dropout grew out of work in Hinton's lab at the University of Toronto in 2011 and 2012. Hinton has often described the inspiration coming from analogies to biological evolution and to bank fraud prevention, where rotating bank tellers reduces opportunities for collusion. Translated to neural networks, randomly removing units during training prevents groups of neurons from forming brittle conspiracies that fail when the input distribution shifts.
The technique was first publicly presented in a 2012 arXiv preprint by Hinton, Srivastava, Krizhevsky, Sutskever, and Salakhutdinov titled "Improving neural networks by preventing co-adaptation of feature detectors," submitted on July 3, 2012 [1]. That paper showed substantial gains on TIMIT phone recognition, CIFAR-10, MNIST, and Reuters document classification, and it was discussed at the NIPS 2012 deep learning workshop. The same year, Krizhevsky, Sutskever, and Hinton used dropout in AlexNet to win the ILSVRC ImageNet competition by a wide margin [11]. AlexNet achieved a top-5 error of 15.3 percent versus 26.2 percent for the second-place entry, a gap of nearly 11 percentage points that made dropout famous overnight in the computer vision community [11].
The full theoretical and empirical treatment was published in 2014 in the Journal of Machine Learning Research as "Dropout: A Simple Way to Prevent Neural Networks from Overfitting" [2]. That paper has since accumulated more than 42,000 citations on Semantic Scholar [19] and is one of the most cited works in machine learning. It established dropout as a default ingredient of deep model training for nearly a decade, until very large data and architectural changes began to reduce its perceived necessity.
The core mechanism of dropout is straightforward. During each forward pass in training, each neuron in the dropout layer is independently zeroed out with probability p (typically called the dropout rate). The remaining active neurons have their outputs scaled by 1/(1-p) to maintain the expected sum of activations. During inference (test time), all neurons are active and no scaling is needed.
Consider a layer with output activations h = [h_1, h_2, ..., h_n]. During training with dropout rate p:
In equation form, the forward pass for a layer becomes y = f(W (x ⊙ m)) where m is the binary Bernoulli mask, ⊙ is element-wise multiplication, and f is the activation function. The mask is resampled at every training step, so a different random subset of neurons is active each time. This means the network effectively trains a different "thinned" sub-network on each mini-batch.
Without any rescaling, the expected value of a neuron's output drops by a factor of (1 - p) when dropout is active. If we want the expected output to match what a network without dropout would produce, we have two choices: scale at test time or scale at training time. Inverted dropout multiplies the surviving activations by 1/(1-p) so that E[h_out] = h, which keeps the inference graph identical to a normal forward pass. The original 2014 paper described the opposite convention (scale weights by (1 - p) at test time), but inverted dropout is now universal in PyTorch, TensorFlow, JAX, Keras, and other modern frameworks.
At test time, all neurons are active. If using inverted dropout, no modification is needed because the scaling was already applied during training. If using the original formulation, the weights coming out of a dropout layer must be multiplied by (1 - p) at test time to compensate for the fact that more neurons are active than during any single training step.
| Approach | During training | During inference |
|---|---|---|
| Standard dropout | Zero out neurons with probability p, no scaling | Multiply outgoing weights by (1 - p) |
| Inverted dropout | Zero out neurons with probability p, scale survivors by 1/(1-p) | No modification needed |
Inverted dropout is preferred in practice because it keeps the test-time forward pass identical to a normal forward pass, which simplifies deployment, model export, and inference optimizations like graph fusion or quantization. All major deep learning frameworks (PyTorch, TensorFlow, JAX) implement inverted dropout.
Dropout's effectiveness has been explained through several complementary perspectives. None of these is wrong, and most authors view them as different views of the same phenomenon.
The original motivation from Hinton et al. [1] was to prevent "co-adaptation" of feature detectors. As the 2012 preprint puts it, dropout "prevents complex co-adaptations in which a feature detector is only helpful in the context of several other specific feature detectors" [1]. In a standard neural network, neurons can develop complex co-dependencies where one neuron's useful representation depends on the specific outputs of several other neurons. If those partner neurons are absent at test time (because the input distribution has shifted slightly), the co-adapted feature becomes unreliable. By randomly removing neurons during training, dropout forces each neuron to learn features that are useful in combination with many different random subsets of other neurons, producing more robust individual features.
A network with n neurons that can be dropped has 2^n possible thinned sub-networks (each corresponding to a different dropout mask). Training with dropout can be viewed as training an exponentially large ensemble of these sub-networks simultaneously, with shared weights. At test time, using the full network with scaled weights approximates the geometric mean of the predictions of all these sub-networks [2]. Ensembles are well known to reduce variance and improve generalization, and dropout provides a computationally cheap way to approximate ensemble averaging without ever instantiating the full ensemble.
For a linear model with squared loss, this approximation is exact: the deterministic test-time prediction equals the expectation over masks. For nonlinear networks the test-time output is only an approximation of the true ensemble mean, but it works remarkably well in practice.
Dropout injects multiplicative noise into the hidden representations during training. This noise acts as a regularizer by preventing the network from relying too heavily on any particular activation pattern. For squared-error linear regression, adding multiplicative noise with mean 1 is equivalent to adding an L2 penalty whose strength depends on p, the input variance, and the weight magnitudes. This connection makes dropout closely related to weight decay and helps explain why combining the two can sometimes overshoot the optimal regularization strength.
Srivastava et al. observed that networks trained with dropout tend to develop activations that are more sparse and less correlated than networks trained without dropout [2]. Each unit is forced to be useful on its own, which encourages it to detect a recognizable feature rather than fragments of features that only become meaningful in combination with specific neighbors. The result is reminiscent of biological neural systems, where neurons tend to fire sparsely.
Yarin Gal and Zoubin Ghahramani showed that training with dropout is mathematically equivalent to approximate variational inference in a deep Gaussian process [3]. Their 2016 ICML paper is described as "casting dropout training in deep neural networks (NNs) as approximate Bayesian inference in deep Gaussian processes" [3]. Under this interpretation, the dropout mask samples correspond to samples from the approximate posterior distribution over the network's weights. This connection provides a theoretical foundation for using dropout not only as a regularizer but also as a tool for uncertainty estimation (see Monte Carlo dropout below) and ties dropout to the broader literature on Bayesian neural networks.
The dropout rate p (the probability of zeroing out a neuron) is the main hyperparameter for dropout. The optimal rate depends on the layer type, the network architecture, and the dataset size.
| Layer type | Recommended dropout rate | Rationale |
|---|---|---|
| Input layer | 0.0 to 0.2 | Dropping too many input features discards information; light dropout can help |
| Hidden layers (fully connected) | 0.5 | Original recommendation from Hinton et al.; balances regularization and capacity |
| Convolutional layers | 0.1 to 0.3 | Spatial redundancy in conv layers means less dropout is needed |
| Recurrent layers | 0.2 to 0.5 | Applied to non-recurrent connections; recurrent connections may use variational dropout |
| Transformer attention and FFN | 0.1 (typical) | Lower rates work well with the residual structure |
| Output layer | 0.0 | Dropping output neurons distorts the loss signal |
The rate of 0.5 for hidden layers was the original recommendation from the 2012 and 2014 papers and has a nice property: it maximizes the variance of a Bernoulli random variable, which maximizes the noise injected per neuron. In practice, modern architectures often use lower rates (0.1 to 0.3) because other forms of regularization are also applied.
Larger networks typically benefit from higher dropout rates because they have more capacity to overfit. Smaller networks may not tolerate high dropout rates because too much capacity is being removed at each training step. When the training set is very large relative to the model capacity, dropout becomes less necessary because overfitting is less of a concern.
| Model | Year | Dropout setting |
|---|---|---|
| AlexNet [11] | 2012 | 0.5 in fully connected layers |
| VGG-16 | 2014 | 0.5 in fully connected layers |
| Original Transformer [9] | 2017 | 0.1 attention, 0.1 residual, 0.1 embedding |
| BERT-Base | 2018 | 0.1 across attention and FFN |
| GPT-2 | 2019 | 0.1 |
| GPT-3 [10] | 2020 | 0.0 to 0.1 depending on model scale |
| PaLM | 2022 | 0.0 |
| Llama, Llama 2 | 2023 | 0.0 (no dropout in pretraining) [12] |
| ViT-B/16 | 2020 | 0.0 (default in JAX/Flax reference impl) |
| DeiT-B | 2020 | Stochastic depth, no standard dropout in attention |
| ConvNeXt | 2022 | Stochastic depth, no standard dropout |
The pattern is clear: as scale grew and training data exceeded model capacity, dropout rates dropped, often to zero. For smaller architectures or fine-tuning on limited downstream data, the older 0.1 to 0.5 ranges still apply.
Dropout is not always helpful. There are several situations where it can hurt accuracy or interact badly with other techniques.
A useful diagnostic: if training loss and validation loss are close, dropout is likely unnecessary or even harmful. Dropout earns its keep when training loss is much lower than validation loss, which is the classic overfitting signal.
Since the original dropout paper, researchers have developed many specialized variants for different architectures and settings. Several have become more common in modern practice than the original.
DropConnect (Wan et al., 2013) is a generalization of dropout that randomly zeros out individual weights rather than neuron activations [4]. While dropout sets entire neuron outputs to zero (dropping all outgoing connections from a neuron), DropConnect sets individual connections to zero. This provides a finer-grained form of regularization. DropConnect can be more effective than dropout in some settings, but it is also more computationally expensive because the dropout mask is applied to the weight matrix, which is much larger than the activation vector.
Standard dropout drops individual activations, which is appropriate for fully connected layers but less effective for convolutional layers. In a convolutional neural network, adjacent spatial positions in a feature map are highly correlated, so dropping individual pixels has little effect because the surrounding pixels carry redundant information.
Spatial dropout (Tompson et al., 2015) addresses this by dropping entire feature maps (channels) rather than individual activations [5]. If a feature map is dropped, all spatial positions in that map are set to zero. This forces the network to learn redundant representations across different channels rather than relying on any single feature map. PyTorch implements this as nn.Dropout2d and TensorFlow as SpatialDropout2D.
DropBlock (Ghiasi et al., 2018) extends spatial dropout by dropping contiguous rectangular regions within feature maps [6]. Rather than dropping an entire channel (which can be too aggressive) or individual pixels (which is too weak), DropBlock drops a block of size block_size by block_size. The block size is a hyperparameter that controls the granularity of the dropped region. When block_size = 1, DropBlock reduces to standard dropout; when block_size covers the entire feature map, it reduces to spatial dropout.
DropBlock was shown to be particularly effective for training object detection and semantic segmentation models, where spatial structure is important and the network needs to be regularized at the right spatial scale.
The original dropout paper noted that naive application of dropout in recurrent neural networks is harmful, because resampling the mask at every time step injects noise into the recurrent connections and disrupts long-range memory. Gal and Ghahramani (2016) introduced "variational dropout" for RNNs, which uses the same dropout mask at every time step within a sequence (per layer) [3]. This provides regularization without destroying the recurrent signal and quickly became the standard way to apply dropout in LSTM and GRU language models.
A different and somewhat overlapping line of work, also called variational dropout, treats the dropout rate itself as a learned parameter via variational inference. Kingma, Salimans, and Welling (2015) and Molchanov, Ashukha, and Vetrov (2017) frame dropout as variational Bayes with Gaussian noise and optimize the per-weight noise level [7]. Many weights converge to a dropout rate near 1, effectively pruning them from the network. This produces extreme sparsity (often 90% or more) with minimal accuracy loss and connects dropout directly to the literature on neural network compression.
Concrete dropout (Gal, Hron, and Kendall, 2017) uses a continuous relaxation of the Bernoulli distribution (the Concrete or Gumbel-Softmax distribution) to make the dropout rate differentiable [13]. With this relaxation, the dropout probability can be optimized directly by gradient descent rather than tuned by grid search. This is particularly valuable when dropout is used for uncertainty estimation, since well-calibrated uncertainty depends on choosing the right rate. Empirically, concrete dropout learns higher dropout rates for the second and final layers of small image classifiers and lower rates for the input and middle layers, a pattern that is hard to discover by hand.
Stochastic depth, introduced by Huang, Sun, Liu, Sedra, and Weinberger in 2016, drops entire residual blocks during training rather than individual neurons [14]. For each mini-batch, each residual block is replaced by an identity function with some probability, which means the network trains short paths and uses long paths at test time. Drop probabilities are usually set to increase linearly with depth so that earlier layers are kept and later layers are dropped more often. The same idea, often called DropPath in vision transformer literature, is used in DeiT, Swin Transformer, and ConvNeXt as a primary regularizer in place of standard dropout. The popular timm library implements both DropPath and DropBlock and applies them by default to many modern image models.
Semeniuta, Severyn, and Barth (2016) proposed a different recurrent dropout that drops the hidden state update inside an LSTM or GRU cell rather than the hidden state itself [15]. Because the dropped components are added to the cell state, the long-term memory is preserved and only the additive update is regularized. This works well in practice on language modeling and machine translation benchmarks and complements feed-forward dropout on the input and output of the recurrent layer.
In NLP, word dropout (sometimes called embedding dropout) sets entire word embeddings to zero with some probability before they enter the network. Because the same vector is used everywhere a word appears, dropping the vector once removes that word from the entire input sequence for that forward pass. Embedding dropout is part of the AWD-LSTM language model and several modern text classification setups. It complements positional dropout in transformers, where the sum of token and positional encoding vectors is dropped before the first attention layer.
AlphaDropout (Klambauer, Unterthiner, Mayr, Hochreiter, 2017) is a special dropout designed for self-normalizing neural networks that use the SELU activation [16]. Standard dropout sets values to zero, which destroys the zero-mean unit-variance property that SELU maintains. AlphaDropout sets the dropped values to the negative saturation value of SELU instead and applies an affine transformation that preserves both mean and variance. This keeps the self-normalizing dynamics intact and is implemented as nn.AlphaDropout in PyTorch and tf.keras.layers.AlphaDropout in Keras.
Targeted dropout (Gomez, Zhang, Kamalakara, Madaan, Swersky, Gal, Hinton, 2018) ranks weights or units by importance (typically by magnitude) and applies dropout only to the least important elements [17]. Because dropout already encourages the network to be robust to the absence of dropped components, applying it to the candidate-for-pruning set teaches the network to perform well after those elements are pruned. Targeted dropout consistently outperforms one-shot magnitude pruning at the same sparsity level, often matching the accuracy of an unregularized dense network at half the parameter count.
| Variant | What is dropped | Where it works best | Memory cost | Key advantage |
|---|---|---|---|---|
| Dropout [2] | Individual neuron activations | Fully connected layers | Low | Simple, effective, well-understood |
| DropConnect [4] | Individual weights | Fully connected layers | Higher | Finer-grained regularization |
| Spatial dropout [5] | Entire feature map channels | Convolutional layers | Low | Handles spatial correlation |
| DropBlock [6] | Contiguous spatial regions | Convolutional layers | Low | Tunable spatial granularity |
| Variational dropout (RNN) [3] | Same mask across time steps | LSTMs and GRUs | Low | Preserves long-term memory |
| Variational sparsifying [7] | Learned per-weight noise | Any layer | Moderate | Automatic pruning, sparsification |
| Concrete dropout [13] | Activations with learned rate | Bayesian models | Low | Learnable dropout probability |
| Stochastic depth / DropPath [14] | Entire residual blocks | Deep ResNets, ViTs, ConvNeXt | Low | Trains short, tests deep |
| Recurrent dropout [15] | LSTM hidden state update | LSTMs | Low | Preserves cell memory |
| Word/embedding dropout | Whole word embeddings | NLP models | Low | Token-level regularization |
| AlphaDropout [16] | Activations replaced with SELU saturation value | SELU networks | Low | Preserves self-normalization |
| Targeted dropout [17] | Low-importance weights or units | Pruning pipelines | Low | Yields prunable networks |
Dropout sits in a family of regularizers, and in modern practice it is rarely used alone.
Batch normalization (Ioffe and Szegedy, 2015) and dropout were both introduced as techniques to improve training of deep networks, but they interact in non-obvious ways.
Li et al. (2019) showed that applying dropout before batch normalization can cause a "variance shift" problem [8]. During training, dropout changes the variance of the layer's output (because some neurons are zeroed and others are scaled). Batch normalization computes running statistics of the mean and variance during training and uses these statistics at test time. The training statistics include the variance introduced by dropout, while the test statistics do not (because dropout is disabled at test time). This mismatch between training and test variance can hurt performance.
Several solutions have been proposed:
In practice, many modern convolutional architectures (like ResNet) use batch normalization without dropout and achieve excellent results. The regularization provided by batch normalization, combined with data augmentation, is often sufficient.
Dropout and weight decay both shrink the effective capacity of the model, but they do it differently. Weight decay penalizes the L2 norm of the weights, pulling them toward zero throughout training. Dropout injects noise into activations, which encourages each unit to be useful on its own. Used together, they can be complementary, but at high settings they tend to compete: a model with both very strong weight decay and high dropout often underfits. A common pattern is to use weight decay around 1e-4 to 1e-2 with dropout around 0.1 to 0.3 in convolutional and transformer models.
Data augmentation attacks overfitting from the input side by showing the model many transformed versions of each example. Dropout attacks it from the inside by perturbing hidden activations. They address different failure modes and combine well, which is why most successful image classification recipes use both. For language models, dropout substitutes for the role data augmentation plays in vision because text is harder to augment without changing meaning.
Label smoothing softens the one-hot training labels by mixing in a small amount of uniform probability. It penalizes overconfident predictions, which is a different lever from dropout's noise injection. Modern transformer and vision pipelines often use both at modest settings, for example dropout 0.1 with label smoothing 0.1.
Dropout's connection to ensembles is real but approximate. Combining dropout with explicit ensembling (training several independently initialized models and averaging their outputs) usually still improves accuracy, which suggests dropout does not fully capture the variance reduction of true ensembling.
The role of dropout has evolved significantly as neural network architectures have changed.
During the era of AlexNet, VGGNet, and similar architectures, dropout was essential. These networks had large fully connected layers at the end (sometimes with tens of millions of parameters), which were highly prone to overfitting. Without dropout, these models would severely overfit on datasets like ImageNet. Dropout rates of 0.5 were standard in these fully connected layers.
As architectures shifted toward global average pooling (eliminating the large fully connected layers), the role of dropout diminished in convolutional networks. ResNet, for example, does not use dropout in its standard configuration. Batch normalization and data augmentation provided sufficient regularization. When dropout was used in convolutional networks, it was typically spatial dropout or DropBlock rather than standard dropout.
Normalizer-Free Networks (NFNets) by Brock, De, Smith, and Simonyan (2021) deliberately removed batch normalization and instead leaned heavily on dropout, gradient clipping, and stochastic depth to keep large image models stable [18]. This is one of the few recent settings where dropout was explicitly turned up rather than down.
In transformer architectures, dropout is used but in a more targeted and restrained manner. The original transformer paper (Vaswani et al., 2017) applied dropout in three places [9]:
For large language models with billions of parameters trained on trillions of tokens, dropout is often reduced or eliminated entirely. The sheer volume of training data provides implicit regularization, and the model is far from overfitting on the training set. GPT-3, for example, used dropout rates of 0.0 to 0.1 depending on model size [10]. PaLM and Chinchilla report using no dropout at all during pre-training. Llama and Llama 2 also use zero dropout during pretraining [12], a choice that is now standard for frontier-scale single-epoch pretraining.
However, dropout remains important during fine-tuning, where a large pre-trained model is adapted to a smaller downstream dataset. In this setting, overfitting is a real risk, and dropout rates of 0.1 to 0.3 are commonly applied. The default attention_dropout in many Hugging Face fine-tuning recipes is set to a small nonzero value, while the same parameter is zero in the base pretraining configuration.
Vision Transformers (ViT) usually disable hidden_dropout_prob and attention_probs_dropout_prob in their reference Flax implementation and rely on stochastic depth, RandAugment, and Mixup for regularization. DeiT (Touvron et al., 2021) made this combination popular, using DropPath with a probability that increases linearly across the depth of the network rather than standard dropout. Steiner et al. (2021) studied the trade-off between data augmentation and model regularization in detail and found that augmentation usually pays off more than dropout for ViT-scale models. ConvNeXt and modern hybrid architectures follow the same recipe: stochastic depth instead of unit dropout, plus heavy data augmentation.
One of the most influential extensions of dropout is Monte Carlo (MC) dropout, proposed by Yarin Gal and Zoubin Ghahramani in 2016 [3]. MC dropout provides a practical method for estimating uncertainty in neural network predictions without modifying the model architecture or training procedure.
The key insight is simple: keep dropout enabled at test time, run multiple forward passes with different dropout masks, and treat the variation in predictions as a measure of uncertainty.
Specifically:
Gal and Ghahramani proved that this procedure is mathematically equivalent to performing approximate variational inference in a deep Gaussian process, where the dropout distribution serves as an approximate posterior over the model weights [3].
MC dropout has found applications in domains where knowing the model's confidence is as important as the prediction itself:
| Application domain | How uncertainty is used |
|---|---|
| Medical diagnosis | Flag low-confidence predictions for human review |
| Autonomous vehicles | Increase caution when perception uncertainty is high |
| Active learning | Select the most uncertain samples for labeling |
| Bayesian optimization | Balance exploration and exploitation using uncertainty |
| Anomaly detection | High uncertainty on unusual inputs signals potential anomalies |
The main cost of MC dropout is the need for multiple forward passes at test time. Typical choices of T range from 10 to 100 passes. This increases inference latency linearly, which may be unacceptable for real-time applications. Several methods have been proposed to reduce this cost, including learned uncertainty estimates, deep ensembles, single-pass approximations, and last-layer Bayesian methods.
The quality of the uncertainty estimates depends on the dropout rate and network architecture. Gal and Ghahramani recommend tuning the dropout rate as a model hyperparameter (it corresponds to the prior length-scale in the Gaussian process interpretation) rather than treating it purely as a regularization parameter. Concrete dropout was developed in part to make this tuning automatic [13].
Dropout is built into every major deep learning framework. The basic API is similar across frameworks: instantiate a dropout layer with a rate, place it in your model, and switch the model between training and evaluation modes.
PyTorch makes dropout straightforward:
import torch
import torch.nn as nn
class MyModel(nn.Module):
def __init__(self):
super().__init__()
self.fc1 = nn.Linear(784, 512)
self.dropout1 = nn.Dropout(p=0.5)
self.fc2 = nn.Linear(512, 256)
self.dropout2 = nn.Dropout(p=0.5)
self.fc3 = nn.Linear(256, 10)
def forward(self, x):
x = torch.relu(self.fc1(x))
x = self.dropout1(x)
x = torch.relu(self.fc2(x))
x = self.dropout2(x)
x = self.fc3(x)
return x
model = MyModel()
model.train() # dropout is active
model.eval() # dropout is bypassed
Note that nn.Dropout in PyTorch uses inverted dropout by default and is automatically disabled when model.eval() is called. PyTorch also offers nn.Dropout1d, nn.Dropout2d, and nn.Dropout3d for spatial dropout on 1D, 2D, and 3D feature maps respectively, plus nn.AlphaDropout for SELU networks. For MC dropout, you would keep the model in training mode (model.train()) or manually enable dropout during inference by toggling each dropout layer's training flag.
Keras provides tf.keras.layers.Dropout, tf.keras.layers.SpatialDropout1D, tf.keras.layers.SpatialDropout2D, tf.keras.layers.SpatialDropout3D, and tf.keras.layers.AlphaDropout. The Dropout layer takes a rate argument that is the probability of dropping a unit (the same convention as PyTorch). The training argument to model(inputs, training=True) controls whether dropout is active. Keras also automatically toggles dropout based on whether you call model.fit (training) or model.predict (inference).
In JAX-based libraries, dropout is implemented as a deterministic function of an input tensor and a PRNG key. Flax provides flax.linen.Dropout(rate=0.5) and Haiku provides haiku.dropout(rng, rate, x). Because JAX is functional, the user must explicitly pass a PRNG key to generate the dropout mask, and the layer must be told whether it is in training mode through a deterministic flag (in Flax) or by skipping the call entirely at inference (in Haiku).
The common pattern is the same in all three frameworks: dropout is an in-place perturbation applied during the forward pass, scaled to preserve expectation, and disabled at inference. The framework-specific differences are about how training versus inference mode is signaled and how randomness is plumbed.
A few common pitfalls:
model.eval() in PyTorch before inference, which leaves dropout active and produces noisy, inconsistent predictions.The Srivastava et al. 2014 paper [2] reports a series of ablations that established dropout's value across domains:
The paper also showed that the hidden representations learned with dropout are sparser than those learned without it and that the learned weights are smaller in magnitude, which is consistent with the implicit-L2 view of dropout.
Dropout was a pivotal contribution to deep learning, and its influence shows up in several distinct ways.
Practical impact. Dropout directly enabled training deeper and larger networks on the datasets available in the early 2010s. AlexNet (Krizhevsky, Sutskever, and Hinton, 2012), which kicked off the deep learning revolution by winning the 2012 ImageNet competition, relied heavily on dropout to prevent overfitting [11]. Without dropout, AlexNet's large fully connected layers would have overfit catastrophically. Dropout remained a default ingredient in image classification, speech recognition, and language modeling pipelines for nearly a decade.
Conceptual impact. Dropout introduced the idea that noise during training can be beneficial, not just tolerable. This concept influenced many subsequent techniques: data augmentation strategies, label smoothing, stochastic depth, mixup, cutmix, and the noise mechanisms in diffusion models. The view of training as implicit ensembling also influenced model averaging methods like SWA and stochastic weight averaging.
Theoretical impact. The connection between dropout and Bayesian inference, formalized by Gal and Ghahramani, opened a bridge between deep learning and probabilistic modeling. It enabled practical uncertainty estimation in deep networks and motivated a wave of work on Bayesian deep learning, last-layer Bayesian methods, and deep ensembles.
The Srivastava et al. 2014 JMLR paper [2] has been cited more than 42,000 times according to Semantic Scholar [19], placing it among the most cited papers in all of machine learning. While dropout's relative importance has decreased in the era of massive datasets and large language models (where the data itself provides regularization), it remains an essential tool for smaller-scale problems, fine-tuning, and uncertainty estimation. In modern practice the spirit of dropout lives on through stochastic depth, DropPath, DropBlock, and the various noise injection schemes in self-supervised and generative models.
What is dropout in a neural network? Dropout is a regularization method that randomly sets a fraction p of neuron outputs to zero on each training step, preventing neurons from co-adapting and reducing overfitting [2]. At inference, all neurons are used and the math is rescaled so the expected output matches training.
Who invented dropout and when? Dropout was introduced by Geoffrey Hinton, Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov in a July 2012 arXiv preprint [1] and fully described in their 2014 JMLR paper [2].
What is a good dropout rate? A rate of 0.5 was the original recommendation for fully connected hidden layers [2], but modern convolutional and transformer models typically use 0.1 to 0.3, and frontier-scale language model pretraining often uses 0.0 [12].
Is dropout used during inference? No. Standard dropout is disabled at test time, when the full network is used. The one exception is Monte Carlo dropout, which deliberately keeps dropout on at inference to estimate uncertainty [3].
Why do large language models often use no dropout? When training data vastly exceeds model capacity (trillions of tokens, a single pass), the model is not at risk of memorizing the data, so dropout provides little benefit and can slow convergence. Llama and Llama 2 use zero dropout in pretraining [12].