ReLU

ReLU, short for rectified linear unit, is an activation function that maps a real input to its positive part:

$$\operatorname{ReLU}(x) = \max(0,x).$$

Applied inside a neural network, it preserves a positive pre-activation and replaces a non-positive one with zero. ReLU is continuous, inexpensive to evaluate, and capable of producing exact-zero activations. It became especially influential through deep rectifier experiments in 2010 and 2011 and through the 2012 AlexNet system.^[3][4][5] Those papers established important empirical results, but ReLU did not by itself make deep learning possible or solve every vanishing-gradient problem. Its behavior depends on the surrounding weights, initialization, architecture, data, optimizer, and implementation.

Definition

The scalar function can be written piecewise:

$$\operatorname{ReLU}(x) = \begin{cases} 0, & x \le 0, \\ x, & x > 0. \end{cases}$$

For a vector, matrix, or higher-order tensor, ReLU is normally applied element by element. This is also the operator semantics specified by ONNX and documented by TensorFlow.^[22][25] If an affine layer computes

$$z = Wx + b,$$

then a ReLU hidden layer returns

$$h = \operatorname{ReLU}(z)$$

elementwise. The affine transformation mixes features; ReLU supplies the nonlinearity by selecting which coordinates are active.

For inputs away from zero, the derivative is

$$\frac{d}{dx}\operatorname{ReLU}(x) = \begin{cases} 0, & x < 0, \\ 1, & x > 0. \end{cases}$$

The ordinary derivative does not exist at zero because the left and right derivatives differ. In software, the backward rule must nevertheless choose a value there. A common convention is zero; JAX documents that choice explicitly.^[23] Choosing a particular derivative at this one point does not make the mathematical function differentiable there. It defines the rule used by that implementation when a pre-activation is exactly zero.

Several elementary properties follow from the definition:

• Continuity: the two pieces meet at the origin.
• Convexity: the graph is the maximum of the affine functions zero and $x$.
• Positive homogeneity: for $c \ge 0$, $\operatorname{ReLU}(cx)=c\operatorname{ReLU}(x)$.
• Monotonicity: increasing the input cannot decrease the output.
• One-Lipschitz scalar map: $|\operatorname{ReLU}(x)-\operatorname{ReLU}(y)| \le |x-y|$.
• Unbounded positive branch: there is no upper saturation value.

The word "linear" in the name does not mean that a ReLU network is globally linear. A composition of affine maps and ReLUs is continuous and piecewise affine: fixing the on/off pattern of all units gives one affine map, while changing that pattern moves to another region. Research on deep rectifier networks formalizes how depth can create many such input regions.^[11] The number of regions is an expressivity measure under stated architectural assumptions, not a direct measure of accuracy or generalization.

Historical development

ReLU has a longer prehistory than the shorthand account that begins in 2010. In 1969, Kunihiko Fukushima published a multilayer visual-feature network built from analog threshold elements.^[1] A 2000 Nature paper by Richard Hahnloser and colleagues reported digital selection and analogue amplification in a cortex-inspired silicon circuit.^[2] These are relevant antecedents, but the sources do not justify a simple claim that one of them exclusively "invented ReLU" in its modern deep-learning role.

Vinod Nair and Geoffrey Hinton's 2010 ICML paper brought rectified units into a specific probabilistic-learning setting. It represented an integer-valued unit as many binary units with shared weights and shifted biases, whose expected total activity approximates the softplus function. For efficient sampling, the authors used a noisy rectified approximation. Their experiments compared those noisy rectified hidden units with binary hidden units in restricted Boltzmann machines, followed by discriminative fine-tuning on object recognition and face verification tasks.^[3] The paper therefore did more than insert a deterministic $\max(0,x)$ into an ordinary feed-forward classifier.

In 2011, Xavier Glorot, Antoine Bordes, and Yoshua Bengio evaluated deterministic rectifier units in deep supervised networks. Their experiments reported equal or better results than tanh networks on the tasks studied and showed that the rectifier models could reach their best performance without unsupervised pretraining on the large labeled tasks they evaluated.^[4] This was evidence about particular datasets and training procedures, not a proof that unsupervised learning was generally unnecessary.

AlexNet made ReLU highly visible in 2012. Its authors applied ReLU after every convolutional and fully connected hidden layer. Their frequently repeated speed result came from a controlled but narrow comparison: on a particular four-layer convolutional network for CIFAR-10, the ReLU version reached 25 percent training error in one-sixth the iterations required by an equivalent tanh version. The learning rates were chosen separately and neither comparison model used regularization. The paper itself noted that the magnitude of the effect varied with architecture.^[5] It is therefore inaccurate to turn this result into a universal claim that ReLU trains every network six times faster.

ReLU then became a standard component of influential convolutional architectures. VGG used ReLU in all hidden layers, while the original ResNet designs placed ReLU within residual blocks.^[12][13] Its role was important, but those systems also depended on convolutional design, data, optimization, normalization, residual connections, and hardware.

Optimization behavior

Active paths and gradient flow

Sigmoid and tanh activations can enter regions where their derivatives become small. ReLU has derivative one on its positive branch, so the activation itself does not shrink a backpropagated signal along an active path. Glorot and colleagues used this observation to motivate rectifier networks.^[4] It is a local statement. A deep-network gradient also contains weight matrices, normalization operations, residual paths, and other activation gates. Negative ReLU pre-activations contribute derivative zero, and products involving the weights can still vanish or explode.

This distinction corrects the claim that ReLU "solved" vanishing gradients. Earlier work connected difficult deep-network optimization to activation saturation, the scale of activations and gradients, and the singular values of layer Jacobians.^[6] Later theoretical work on ReLU networks likewise identified both vanishing and exploding activation-length failure modes and showed that the relevant conditions depend on initialization and architecture.^[10] ReLU removes one source of saturation on the positive side; it does not guarantee a well-conditioned network.

Initialization

A widely used rectifier-aware initialization samples zero-mean weights with variance proportional to $2/\text{fan-in}$ for an ordinary ReLU layer. He and colleagues derived this scale while accounting for the one-sided rectifier and used it to train substantially deeper rectifier models from scratch.^[8] Hanin and Rolnick later proved, within their model assumptions, that variance $2/\text{fan-in}$ prevents the mean activation length from growing or shrinking exponentially in fully connected and convolutional ReLU networks. They also showed that residual branches require appropriate scaling and that controlling the mean does not by itself control every variance failure mode.^[10]

This is why "use He initialization" is useful but incomplete advice. Fan mode, weight layout, bias choice, residual scaling, normalization, width, and the exact activation all matter. PyTorch's initialization API, for example, distinguishes fan-in from fan-out modes and accepts the intended nonlinearity when calculating gains.^[24] A framework default is not evidence that every ReLU architecture has the same optimal initialization.

Nondifferentiability at zero

The corner at zero rarely creates a practical optimization obstacle by itself. Gradient-based software uses a defined backward convention, and exact equality to zero is only one boundary between two regions. The more consequential issue is the entire negative half-line, where the local derivative is zero. Smoothness and gradient availability are separate questions: a smooth activation may have a very small derivative, while ReLU has a full derivative of one on its positive side and zero on its negative side.

Sparsity and computation

ReLU produces an exact zero whenever a pre-activation is non-positive. This can make a representation sparse, but the fraction of zeros is not a universal constant. It varies with the input distribution, learned biases and weights, normalization, architecture, layer, and stage of training. In the 2011 experiments, roughly half the hidden activations were zero just after the paper's stated uniform initialization, and additional sparsity-inducing regularization could raise that fraction.^[4] That observation should not be generalized into a claim that every ReLU layer is 50 to 80 percent inactive.

Exact zeros also do not automatically make ordinary dense execution faster. A dense matrix multiplication normally processes the full tensor unless the runtime uses a sparse representation, predicts which channels are inactive, or employs a kernel that skips the corresponding work and memory movement. ReLU itself is a simple comparison-like elementwise operation, but the following dense layer is often much more expensive.

Recent language-model work illustrates both the opportunity and the qualification. A 2024 ICLR study reported little convergence or performance difference among the activation choices in its tested language-model settings and proposed ReLU-based schemes that reduced estimated inference computation by as much as threefold. Those savings relied on exploiting structured activation sparsity and, in some experiments, adding or shifting rectifiers; they are not the cost of merely replacing one function call.^[19]

The TEAL study at ICLR 2025 achieved up to 1.53-fold and 1.8-fold wall-clock decoding speedups at 40 and 50 percent model-wide sparsity, respectively, using magnitude pruning and specialized sparse kernels.^[20] A separate ICML 2025 study found, in its decoder-only experiments, that ReLU-trained models became sparser with more training data and were sparser than matched SiLU models at comparable reported performance. Its 2.4-billion-parameter demonstration reached a measured 93.52 percent sparsity under the paper's metric and 4.1-fold speedup over its dense implementation.^[21] These results show that activation sparsity can be useful when model design, sparsity definition, kernels, hardware, and evaluation are aligned. They do not establish a hardware-independent ReLU speedup.

The related claim that ReLU sparsity automatically regularizes a model or prevents overfitting is also too broad. Sparse intermediate values can be an architectural property, but generalization is affected by the full training procedure. The 2011 rectifier paper discussed advantages and hypotheses about sparsity; it did not prove a universal regularization effect.^[4]

Dying ReLU

A ReLU unit is inactive on an input when its pre-activation is non-positive. If a unit remains strictly negative for every relevant training example, the local ReLU derivative is zero for each one, so the loss supplies no gradient through that gate to the unit's incoming weight and bias. This condition is commonly called a dead ReLU.

"Dead" should not be treated as a universal mathematical guarantee of permanence. Inputs to the unit can change when earlier layers are updated through other paths, data can change, and an optimizer can alter parameters for reasons other than that unit's current local gradient. The operational concern is that a unit that stays on the negative side contributes no activation and receives no ordinary gradient through its own gate.

Theoretical results require even more precise wording. Lu and colleagues studied a worst case in which an entire fully connected ReLU network is constant at initialization. Under the distributions and fixed-width architectures in their theorems, the probability of this "born dead" event approaches one as depth tends to infinity. Their analysis also found that increasing width can reduce the event probability at a fixed depth and proposed randomized asymmetric initialization.^[9] The result does not say that death increases with both width and depth, nor does it establish that a fixed percentage of units dies in every practical model.

Useful diagnostics distinguish several cases:

• the fraction of activations equal to zero on a representative dataset;
• units that are inactive for every sample in that dataset;
• pre-activation distributions that sit just below zero versus far below it;
• zero gradients caused by the ReLU gate versus small gradients elsewhere;
• a unit-level inactivity problem versus collapse of an entire network.

Possible responses include checking the learning rate and input scale, revisiting bias and weight initialization, examining normalization and residual scaling, or testing an activation with a nonzero negative slope. None is a guaranteed cure, and changing an activation can alter a pretrained model's behavior.

Common variants

Rectifier variants change the negative branch, cap the positive branch, or change the power applied after thresholding.

Scroll sideways for more →

Leaky ReLU and PReLU

Leaky ReLU keeps a fixed slope on the negative side, so negative pre-activations can pass a nonzero local gradient. The 2013 acoustic-model paper used slope $0.01$ and evaluated ReLU and leaky ReLU in a specific speech-recognition system.^[7] Values used elsewhere are hyperparameters, not part of one universal definition.

PReLU makes the negative slope learnable. He and colleagues considered channel-wise and channel-shared forms. Their paper also reported 4.94 percent top-5 test error for a multi-model ImageNet entry, compared with 6.66 percent for the cited ILSVRC 2014 winner, and described the result as the first reported one below a cited 5.1 percent human-level estimate.^[8] That historical result belongs to the complete system and evaluation protocol. It should not be stated as though PReLU alone guaranteed the improvement.

ELU and SELU

ELU approaches $-\alpha$ for large negative inputs rather than remaining flat at zero. Its authors argued that negative outputs can move activation means closer to zero and reported faster learning in their evaluated networks.^[16] These are paper-specific empirical and mechanistic claims, not a general theorem that ELU always trains faster than ReLU.

SELU multiplies an ELU-like function by a fixed $\lambda$ and uses particular constants, approximately $\alpha=1.6733$ and $\lambda=1.0507$. The self-normalizing-network analysis identifies zero mean and unit variance as an attracting fixed point under its assumptions. Those assumptions include a compatible initialization and a feed-forward setting with conditions on the activation distribution.^[17] Ordinary use of the formula does not guarantee self-normalization in every convolutional, recurrent, residual, or normalized network.

Bounded and powered rectifiers

ReLU6 adds an upper cap. MobileNetV2 used ReLU6 in expansion layers and removed nonlinearities from narrow bottleneck outputs to reduce information loss; its authors cited robustness under low-precision computation as the reason for choosing the capped activation.^[18] A cap can help bound values, but it also creates an upper saturated region.

Squared ReLU preserves the exact-zero negative side but changes the positive derivative to $2x$. The Primer architecture-search study identified squared ReLU as one of two principal modifications in its tested Transformer design.^[26] This is evidence for that design and training setup, not a general ranking of activation functions.

ReLU compared with sigmoid, tanh, and GELU

The functions differ in range, saturation, smoothness, and whether they create exact zeros.

Scroll sideways for more →

The table describes the scalar functions, not a universal performance order. ReLU's positive branch avoids saturation, while its negative branch blocks gradients completely. Sigmoid and tanh retain nonzero but potentially very small derivatives at finite negative inputs. GELU is smooth and allows some negative outputs, but it does not create ReLU's exact half-line of zeros.

Output-layer choice is a separate design question. ReLU constrains an output to be nonnegative but does not normalize it or bound it above. A sigmoid function or softmax may be appropriate when the model output must represent a probability under a corresponding loss and interpretation. Hidden-layer conventions do not determine the output activation automatically.

Use across architectures

ReLU remains historically and practically important in convolutional neural networks. VGG and the original ResNet family are well-known examples, though later variants may change activation placement or type.^[12][13] Reproducing a named architecture requires following its exact specification rather than replacing every nonlinearity based on a generic rule.

ReLU also has a genuine Transformer history. The original 2017 Transformer used a position-wise feed-forward network with two affine transformations and a ReLU between them:

$$\operatorname{FFN}(x)=\max(0,xW_1+b_1)W_2+b_2.$$

This formulation appears in the original paper.^[14] Later Transformer systems made different choices. BERT explicitly used GELU rather than standard ReLU.^[15] Those two primary sources are enough to show that "Transformers use ReLU" and "Transformers replaced ReLU" are both overgeneralizations. Activation choice is an architectural field, and gated feed-forward blocks are not interchangeable with a single pointwise activation.

Contemporary sparse-inference research has renewed interest in ReLU for language models, but the case is conditional. ReLU can expose zeros that a suitable runtime may exploit; smoother or gated alternatives can perform differently under a given training recipe. The correct comparison controls model size, parameterization, training tokens, optimizer, data, evaluation tasks, precision, kernel implementation, and hardware.^[19][20][21]

Implementation and evaluation

The forward operation is straightforward:

for each element x:
    if x > 0:
        y = x
    else:
        y = 0

Official operator and framework documentation agrees on the elementwise $\max(0,x)$ forward definition.^[22][25] Practical differences can still appear around data types, in-place updates, fusion, quantization, and the chosen derivative at zero.

Several checks make ReLU experiments easier to interpret:

• Report whether zeros are counted exactly or by a magnitude threshold.
• Measure sparsity by layer and by representative input, not only as one model-wide average.
• Separate the cost of the ReLU operation from the cost of the following matrix multiplication or convolution.
• Distinguish theoretical operation counts, transferred bytes, kernel latency, end-to-end latency, and throughput.
• State whether sparse kernels or channel-skipping mechanisms are actually enabled.
• Monitor pre-activations as well as post-ReLU values when diagnosing dead units.
• Match the initialization to the activation and the weight-matrix layout.^[8][10][24]
• Preserve the activation used by a pretrained checkpoint unless the change is being evaluated and the model is retrained or adapted appropriately.
• Compare activations under matched architecture, parameter count, training budget, data, and tuning.

ReLU is best understood as a simple operator with specific mathematical consequences, not as an automatic recipe. It offers an unsaturated positive branch, exact-zero outputs, and piecewise-affine structure. The same threshold also creates zero-gradient negative regions, and the same sparsity only saves system work when an implementation exploits it.

See also

• Backpropagation
• Sparsity
• Machine learning terms

References

1. ^Fukushima, K. (1969). "Visual Feature Extraction by a Multilayered Network of Analog Threshold Elements." IEEE Transactions on Systems Science and Cybernetics 5(4), 322-333. doi.org/...TSSC.1969.300225
2. ^Hahnloser, R. H. R., Sarpeshkar, R., Mahowald, M. A., Douglas, R. J., and Seung, H. S. (2000). "Digital Selection and Analogue Amplification Coexist in a Cortex-Inspired Silicon Circuit." Nature 405, 947-951. doi.org/...35016072
3. ^Nair, V., and Hinton, G. E. (2010). "Rectified Linear Units Improve Restricted Boltzmann Machines." Proceedings of the 27th International Conference on Machine Learning, 807-814. cs.toronto.edu/...reluICML.pdf
4. ^Glorot, X., Bordes, A., and Bengio, Y. (2011). "Deep Sparse Rectifier Neural Networks." Proceedings of the 14th International Conference on Artificial Intelligence and Statistics, PMLR 15:315-323. proceedings.mlr.press/...glorot11a
5. ^Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). "ImageNet Classification with Deep Convolutional Neural Networks." Advances in Neural Information Processing Systems 25. papers.neurips.cc/...6b76c8436e924a68c45b-Abstract
6. ^Glorot, X., and Bengio, Y. (2010). "Understanding the Difficulty of Training Deep Feedforward Neural Networks." Proceedings of the 13th International Conference on Artificial Intelligence and Statistics, PMLR 9:249-256. proceedings.mlr.press/...glorot10a
7. ^Maas, A. L., Hannun, A. Y., and Ng, A. Y. (2013). "Rectifier Nonlinearities Improve Neural Network Acoustic Models." ICML Workshop on Deep Learning for Audio, Speech, and Language Processing. ai.stanford.edu/...relu_hybrid_icml2013_final.pdf
8. ^He, K., Zhang, X., Ren, S., and Sun, J. (2015). "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification." Proceedings of ICCV, 1026-1034. openaccess.thecvf.com/...Deep_into_ICCV_2015_paper
9. ^Lu, L., Shin, Y., Su, Y., and Karniadakis, G. E. (2020). "Dying ReLU and Initialization: Theory and Numerical Examples." Communications in Computational Physics 28(5), 1671-1706. doi.org/...cicp.OA-2020-0165
10. ^Hanin, B., and Rolnick, D. (2018). "How to Start Training: The Effect of Initialization and Architecture." Advances in Neural Information Processing Systems 31. proceedings.neurips.cc/...f9f24b15f0e4900-Abstract
11. ^Montufar, G. F., Pascanu, R., Cho, K., and Bengio, Y. (2014). "On the Number of Linear Regions of Deep Neural Networks." Advances in Neural Information Processing Systems 27. proceedings.neurips.cc/...2d96e74617c3d2a-Abstract
12. ^Simonyan, K., and Zisserman, A. (2015). "Very Deep Convolutional Networks for Large-Scale Image Recognition." ICLR 2015. arxiv.org/...1409.1556
13. ^He, K., Zhang, X., Ren, S., and Sun, J. (2016). "Deep Residual Learning for Image Recognition." Proceedings of CVPR, 770-778. openaccess.thecvf.com/..._Learning_CVPR_2016_paper
14. ^Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). "Attention Is All You Need." Advances in Neural Information Processing Systems 30. proceedings.neurips.cc/...bd053c1c4a845aa-Abstract
15. ^Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). "BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding." Proceedings of NAACL-HLT, 4171-4186. aclanthology.org/N19-1423
16. ^Clevert, D.-A., Unterthiner, T., and Hochreiter, S. (2016). "Fast and Accurate Deep Network Learning by Exponential Linear Units." ICLR 2016. arxiv.org/...1511.07289
17. ^Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S. (2017). "Self-Normalizing Neural Networks." Advances in Neural Information Processing Systems 30. proceedings.neurips.cc/...125876103c8f6c4-Abstract
18. ^Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., and Chen, L.-C. (2018). "MobileNetV2: Inverted Residuals and Linear Bottlenecks." Proceedings of CVPR, 4510-4520. openaccess.thecvf.com/...Residuals_CVPR_2018_paper
19. ^Mirzadeh, I., Alizadeh, K., Mehta, S., et al. (2024). "ReLU Strikes Back: Exploiting Activation Sparsity in Large Language Models." ICLR 2024. proceedings.iclr.cc/...b2bd15b-Abstract-Conference
20. ^Liu, J., Ponnusamy, P., Cai, T., Guo, H., Kim, Y., and Athiwaratkun, B. (2025). "Training-Free Activation Sparsity in Large Language Models." ICLR 2025. proceedings.iclr.cc/...1f0b99c-Abstract-Conference
21. ^Luo, Y., Song, C., Han, X., et al. (2025). "Sparsing Law: Towards Large Language Models with Greater Activation Sparsity." Proceedings of ICML, PMLR 267:41311-41330. proceedings.mlr.press/...luo25i
22. ^ONNX. "Relu." Operator specification, version 14. onnx.ai/...onnx__Relu
23. ^JAX documentation. "jax.nn.relu." docs.jax.dev/...jax.nn.relu
24. ^PyTorch documentation. "torch.nn.init." docs.pytorch.org/...nn.init
25. ^TensorFlow documentation. "tf.nn.relu." tensorflow.org/...relu
26. ^So, D. R., Manke, W., Liu, H., Dai, Z., Shazeer, N., and Le, Q. V. (2021). "Searching for Efficient Transformers for Language Modeling." Advances in Neural Information Processing Systems 34. proceedings.neurips.cc/...456e7e51a916ff3-Abstract