Activation Function

An activation function is a mathematical function applied to the output of each neuron (or node) in an artificial neural network. It determines whether a neuron should be "activated" by computing a weighted sum of inputs and mapping the result through a nonlinear transformation. Activation functions are essential to deep learning because they introduce the nonlinearity that allows neural networks to learn complex patterns in data.

Without activation functions, a deep neural network of any depth would reduce to a single linear transformation, regardless of how many layers it contained. This is because the composition of multiple linear functions is itself linear. By inserting a nonlinear activation function after each linear operation, neural networks gain the ability to approximate virtually any continuous function, a result formalized by the Universal Approximation Theorem.

Explain like I'm 5 (ELI5)

Imagine you have a bunch of friends passing notes in a chain. Each friend reads the note, decides how excited they should be about it, and writes a new note based on their excitement level to pass to the next friend. The "excitement rule" each friend uses is like an activation function. If everyone just copied the note exactly (linear), the last friend would get pretty much the same message no matter how many friends were in the chain. But if each friend adds their own twist (nonlinear), the chain can come up with really creative and complicated messages. That twist is what an activation function provides to a neural network.

Why nonlinearity matters

A single layer of a neural network computes a linear transformation of its inputs: y = Wx + b, where W is a weight matrix and b is a bias vector. Stacking two such layers without any activation function gives y = W2(W1x + b1) + b2 = W2W1x + (W2b1 + b2), which is still a linear function of x. No matter how many layers are added, the result is equivalent to one linear layer.

Nonlinear activation functions break this linearity. When a nonlinear function f is applied between layers, the composition f(W2 f(W1 x + b1) + b2) is no longer linear. This allows each layer to learn a different nonlinear transformation of its input, giving the network the representational power to model complex relationships. This property is central to the success of backpropagation in training deep networks, since meaningful gradients can flow through layers that compute genuinely different transformations.

Historical evolution

The history of activation functions mirrors the broader development of neural network research.

Step functions and perceptrons (1950s-1960s)

The earliest artificial neurons, including Frank Rosenblatt's Perceptron (1958), used binary step functions. A step function outputs 1 if the weighted input exceeds a threshold and 0 otherwise. While simple and biologically motivated (neurons either fire or do not), step functions are not differentiable at the threshold, making them incompatible with gradient-based optimization.

Sigmoid and tanh era (1980s-2000s)

The development of the backpropagation algorithm in the 1980s required differentiable activation functions. The sigmoid (logistic) function and the hyperbolic tangent (tanh) became standard choices. These smooth, bounded functions allowed gradient computation throughout the network. However, as networks grew deeper, researchers discovered that both sigmoid and tanh suffer from the vanishing gradient problem: for large or small input values, the gradient approaches zero, effectively halting learning in early layers.

ReLU revolution (2010-2015)

Vinod Nair and Geoffrey Hinton introduced the Rectified Linear Unit (ReLU) in 2010, demonstrating that this simple function could dramatically improve training of deep networks. ReLU's constant gradient for positive inputs eliminated the vanishing gradient problem in one direction and provided computational efficiency. The success of ReLU in AlexNet (2012) for image recognition cemented its dominance. Variants like Leaky ReLU and Parametric ReLU soon followed to address ReLU's limitations.

Modern smooth activations (2016-present)

The rise of transformer architectures brought a new generation of activation functions. The Gaussian Error Linear Unit (GELU), proposed by Hendrycks and Gimpel in 2016, became the default in BERT and GPT models. Google Brain's automated search discovered Swish (SiLU) in 2017. More recently, gated variants like SwiGLU have become standard in large language models such as LLaMA and PaLM.

Common activation functions

Sigmoid (logistic)

The sigmoid function maps any real number to the range (0, 1):

Formula: sigma(x) = 1 / (1 + e^(-x))

Derivative: sigma'(x) = sigma(x) * (1 - sigma(x))

The sigmoid function was historically one of the most popular activation functions. Its output can be interpreted as a probability, making it natural for binary classification output layers. However, sigmoid has several drawbacks for hidden layers. Its outputs are not zero-centered (they range from 0 to 1), which can cause zig-zagging during gradient updates. More critically, the gradient saturates for large positive or negative inputs, approaching zero. This saturation causes the vanishing gradient problem in deep networks, slowing or preventing learning in early layers.

Tanh (hyperbolic tangent)

The tanh function is a rescaled version of sigmoid that maps inputs to the range (-1, 1):

Formula: tanh(x) = (e^x - e^(-x)) / (e^x + e^(-x))

Derivative: tanh'(x) = 1 - tanh(x)^2

Relation to sigmoid: tanh(x) = 2 * sigma(2x) - 1

Tanh has the advantage of being zero-centered, which generally leads to faster convergence during training compared to sigmoid. Despite this improvement, tanh still saturates at extreme values, so it shares the vanishing gradient limitation. Tanh remains common in recurrent neural networks (RNNs) and certain gating mechanisms such as those in LSTMs.

ReLU (Rectified Linear Unit)

ReLU is the most widely used activation function in modern deep learning:

Formula: f(x) = max(0, x)

Derivative: f'(x) = 0 if x < 0, 1 if x > 0 (undefined at x = 0, typically set to 0 or 1)

ReLU has several key advantages. It is computationally cheap, requiring only a comparison operation. For positive inputs, the gradient is always 1, which eliminates the vanishing gradient problem and enables effective training of very deep networks. ReLU also induces sparsity: neurons with negative pre-activation values output exactly zero, meaning a subset of neurons is effectively inactive for any given input. This sparsity can improve computational efficiency and serve as a form of implicit regularization.

The primary limitation is the "dying ReLU" problem. If a neuron's weights are updated such that its input is always negative, the neuron will output zero for every input and receive zero gradient. Once this happens, the neuron can never recover. In practice, this can cause a significant fraction of neurons to become permanently inactive, especially with high learning rates.

Leaky ReLU

Leaky ReLU addresses the dying ReLU problem by allowing a small, non-zero gradient for negative inputs:

Formula: f(x) = x if x > 0, alpha * x if x <= 0 (typically alpha = 0.01)

Derivative: f'(x) = 1 if x > 0, alpha if x <= 0

By using a small positive slope (commonly 0.01) for negative inputs, Leaky ReLU ensures that every neuron always has a non-zero gradient. This means neurons cannot "die" as they can with standard ReLU. The negative slope is a fixed hyperparameter that must be chosen before training.

Parametric ReLU (PReLU)

PReLU generalizes Leaky ReLU by making the negative slope a learnable parameter:

Formula: f(x) = x if x >= 0, alpha * x if x < 0 (alpha is learned during training)

Proposed by He et al. (2015), PReLU allows the network to learn the optimal negative slope through backpropagation. This adds a small number of extra parameters (one per channel or one shared across all channels) but can improve performance. The authors showed that PReLU achieved a then-record result on the ImageNet classification task, surpassing human-level accuracy.

ELU (Exponential Linear Unit)

ELU uses an exponential function for negative inputs, providing a smooth transition:

Formula: f(x) = x if x > 0, alpha * (e^x - 1) if x <= 0 (typically alpha = 1.0)

Derivative: f'(x) = 1 if x > 0, f(x) + alpha if x <= 0

Proposed by Clevert, Unterthiner, and Hochreiter (2015), ELU combines the benefits of ReLU (no vanishing gradient for positive values) with negative outputs that push the mean activation closer to zero. Unlike Leaky ReLU, ELU saturates for large negative values (approaching -alpha), which can make the network more robust to noise. ELU's smoothness at zero provides better gradient flow compared to the sharp corner in ReLU.

SELU (Scaled Exponential Linear Unit)

SELU is a scaled version of ELU with specific parameter values that enable self-normalizing behavior:

Formula: f(x) = lambda * x if x > 0, lambda * alpha * (e^x - 1) if x <= 0

where lambda = 1.0507 and alpha = 1.6733

Introduced by Klambauer et al. (2017), SELU was designed so that activations in a neural network automatically converge toward zero mean and unit variance during forward propagation. The authors proved this self-normalizing property using the Banach fixed-point theorem. SELU eliminates the need for explicit normalization layers like batch normalization in fully connected architectures. However, SELU requires specific weight initialization (LeCun normal) and works best with standard feedforward networks rather than convolutional or recurrent architectures.

GELU (Gaussian Error Linear Unit)

GELU has become the default activation function in transformer models:

Formula: GELU(x) = x * Phi(x) = (x/2) * (1 + erf(x / sqrt(2)))

where Phi(x) is the cumulative distribution function of the standard normal distribution.

Approximation: GELU(x) is approximately equal to 0.5 * x * (1 + tanh(sqrt(2/pi) * (x + 0.044715 * x^3)))

Proposed by Hendrycks and Gimpel (2016), GELU weights inputs by their magnitude using a probabilistic framework. Instead of gating inputs with a hard threshold (as ReLU does) or a fixed function, GELU gates inputs by how much they exceed other values, based on the Gaussian distribution. As x becomes large and positive, GELU approaches the identity function (like ReLU). As x becomes large and negative, it approaches zero. The transition between these two regimes is smooth and non-monotonic; GELU has a slight dip below zero for small negative inputs (its minimum value is approximately -0.17).

GELU gained wide adoption as the activation function in BERT (2018) and the GPT series of models. Its smoothness is believed to benefit optimization in transformer training.

Swish / SiLU (Sigmoid Linear Unit)

Swish was discovered through an automated search by Google Brain researchers:

Formula: f(x) = x * sigma(x) = x / (1 + e^(-x))

where sigma is the sigmoid function.

Derivative: f'(x) = f(x) + sigma(x) * (1 - f(x))

Ramachandran, Zoph, and Le (2017) used reinforcement learning to search over a space of candidate activation functions and found that Swish consistently outperformed ReLU across tasks. Swish is self-gated: the sigmoid component acts as a learned gate on the linear component x. Like GELU, Swish is smooth, non-monotonic, and unbounded above but bounded below (minimum approximately -0.278). Swish has been adopted in architectures like EfficientNet and MobileNetV3. In PyTorch, Swish is available as torch.nn.SiLU.

Mish

Mish is a smooth, self-regularized, non-monotonic activation function:

Formula: f(x) = x * tanh(softplus(x)) = x * tanh(ln(1 + e^x))

Proposed by Diganta Misra (2019), Mish shares several properties with Swish, including smoothness, non-monotonicity, and unboundedness above. Mish showed improvements over both ReLU and Swish in image classification benchmarks such as CIFAR-100 and ImageNet, as well as object detection with YOLOv4. Its self-regularizing property comes from the smooth, continuous first derivative that avoids sharp transitions.

Softmax

Softmax is a multi-input, multi-output activation function used primarily in the output layer for multi-class classification:

Formula: softmax(x_i) = e^(x_i) / sum(e^(x_j)) for all j

Unlike other activation functions that operate element-wise, softmax takes an entire vector of raw scores (logits) and converts them into a probability distribution. Each output is in the range (0, 1) and all outputs sum to 1. Softmax is the standard output layer activation for multi-class classification tasks and is used in the attention mechanism of transformer models to compute attention weights.

Softplus

Softplus is a smooth approximation of ReLU:

Formula: f(x) = ln(1 + e^x)

Derivative: f'(x) = sigma(x) = 1 / (1 + e^(-x))

Softplus produces only positive outputs and is infinitely differentiable, making it useful in contexts where a smooth, strictly positive output is needed. It approaches ReLU as inputs become large and positive but has a smooth curve near zero rather than a sharp corner. Softplus is used as a building block in other activation functions (it appears in the Mish formula) and in variational autoencoders where positive variance parameters are needed.

GLU and SwiGLU (Gated Linear Units)

Gated Linear Units and their variants have become the dominant activation mechanism in modern large language models:

GLU formula: GLU(x, W, V, b, c) = (xW + b) * sigma(xV + c)

where sigma is the sigmoid function and * denotes element-wise multiplication.

SwiGLU formula: SwiGLU(x, W, V, b, c) = (Swish(xW + b)) * (xV + c)

GLU was introduced by Dauphin et al. (2017) for language modeling. The key idea is that one linear projection acts as a "gate" that controls information flow from the other projection. Shazeer (2020) proposed replacing the sigmoid gate with other activation functions, producing variants like SwiGLU (using Swish as the gate) and GeGLU (using GELU).

SwiGLU has been adopted as the feed-forward network activation in LLaMA, Mistral, PaLM, and many other state-of-the-art large language models. Compared to standard GELU-based feed-forward layers, SwiGLU consistently improves perplexity and downstream task performance while requiring an adjusted hidden dimension (typically 2/3 of the original) to keep the parameter count equivalent.

Comparison table

Function	Formula	Output range	Smooth	Monotonic	Zero-centered	Common use cases
Sigmoid	1 / (1 + e^(-x))	(0, 1)	Yes	Yes	No	Binary classification output
Tanh	(e^x - e^(-x)) / (e^x + e^(-x))	(-1, 1)	Yes	Yes	Yes	RNNs, LSTM gates
ReLU	max(0, x)	[0, infinity)	No	Yes	No	CNN hidden layers, default
Leaky ReLU	max(0.01x, x)	(-infinity, infinity)	No	Yes	No	Deep CNNs, GANs
PReLU	max(alpha*x, x), alpha learned	(-infinity, infinity)	No	Yes	No	Image classification
ELU	x if x>0; alpha*(e^x - 1) if x<=0	(-alpha, infinity)	Yes	Yes	Approximately	Deep feedforward networks
SELU	lambda*ELU(x) with fixed lambda, alpha	(-lambda*alpha, infinity)	Yes	Yes	Approximately	Self-normalizing networks
GELU	x * Phi(x)	(~-0.17, infinity)	Yes	No	No	Transformers (BERT, GPT)
Swish/SiLU	x * sigmoid(x)	(~-0.278, infinity)	Yes	No	No	EfficientNet, MobileNet
Mish	x * tanh(softplus(x))	(~-0.31, infinity)	Yes	No	No	YOLOv4, image classification
Softmax	e^(x_i) / sum(e^(x_j))	(0, 1), sums to 1	Yes	N/A	No	Multi-class classification output
Softplus	ln(1 + e^x)	(0, infinity)	Yes	Yes	No	Variance parameters, building block
SwiGLU	Swish(xW+b) * (xV+c)	(-infinity, infinity)	Yes	No	Approximately	LLaMA, PaLM, modern LLMs

How to choose an activation function

Selecting an activation function depends on the network architecture, the task, and the specific layer within the network.

Hidden layers

For most tasks, ReLU is a strong default for hidden layers, especially in convolutional neural networks. If the dying ReLU problem is observed (many neurons producing only zeros), switching to Leaky ReLU or ELU can help. For transformer architectures, GELU is the standard choice. If training a feedforward network without batch normalization, SELU with LeCun normal initialization is worth considering.

Output layers

The output layer activation should match the task:

Task	Recommended output activation	Reason
Binary classification	Sigmoid	Outputs a single probability in (0, 1)
Multi-class classification	Softmax	Outputs a probability distribution over classes
Multi-label classification	Sigmoid (per output)	Each output independently predicts presence of a label
Regression	Linear (no activation)	Outputs can be any real number
Regression (positive only)	Softplus or ReLU	Constrains output to positive values

Architecture-specific conventions

Architecture	Typical activation	Notes
CNNs (ResNet, VGG)	ReLU	Simple, efficient, well-studied
Transformers (encoder, e.g. BERT)	GELU	Standard since BERT (2018)
Transformers (decoder LLMs, e.g. LLaMA)	SwiGLU	Standard since LLaMA (2023)
RNNs / LSTMs	Tanh (hidden), Sigmoid (gates)	Traditional choices for recurrent models
GANs	Leaky ReLU (discriminator), ReLU/Tanh (generator)	Leaky ReLU prevents dead neurons in discriminator
Self-normalizing networks	SELU	Requires LeCun normal initialization

Activation functions in transformers

Transformer architectures use activation functions primarily in their feed-forward network (FFN) sublayers. In the original Transformer paper by Vaswani et al. (2017), the FFN used ReLU: FFN(x) = max(0, xW1 + b1)W2 + b2.

BERT (2018) replaced ReLU with GELU in the FFN, and this choice was carried forward into GPT-2, GPT-3, and many subsequent encoder-based models. The smoother gating behavior of GELU, where inputs near zero are partially rather than fully suppressed, is thought to benefit the optimization dynamics of transformer training.

More recent decoder-only language models have moved to gated FFN variants. LLaMA (2023) introduced the use of SwiGLU in its FFN layers, following Shazeer's (2020) findings that gated activations consistently improved transformer language modeling. The SwiGLU FFN replaces the standard two-matrix FFN with a three-matrix gated structure: FFN_SwiGLU(x) = (Swish(xW1)) * (xW3))W2. This approach has been adopted by Mistral, Gemma, and numerous other model families.

The attention mechanism in transformers also uses softmax to compute attention weights from query-key dot products, though this is typically considered part of the attention computation rather than an activation function in the traditional sense.

Implementation examples

PyTorch

import torch
import torch.nn as nn

# Common activation functions
relu = nn.ReLU()
leaky_relu = nn.LeakyReLU(negative_slope=0.01)
elu = nn.ELU(alpha=1.0)
selu = nn.SELU()
gelu = nn.GELU()
silu = nn.SiLU()  # Swish
mish = nn.Mish()
softmax = nn.Softmax(dim=-1)

# Using in a simple network
model = nn.Sequential(
    nn.Linear(784, 256),
    nn.ReLU(),
    nn.Linear(256, 128),
    nn.ReLU(),
    nn.Linear(128, 10)
)

# Using GELU in a transformer-style FFN
class TransformerFFN(nn.Module):
    def __init__(self, d_model, d_ff):
        super().__init__()
        self.linear1 = nn.Linear(d_model, d_ff)
        self.linear2 = nn.Linear(d_ff, d_model)
        self.activation = nn.GELU()

    def forward(self, x):
        return self.linear2(self.activation(self.linear1(x)))

TensorFlow / Keras

import tensorflow as tf
from tensorflow import keras

# Using activation functions in layers
model = keras.Sequential([
    keras.layers.Dense(256, activation='relu', input_shape=(784,)),
    keras.layers.Dense(128, activation='elu'),
    keras.layers.Dense(10, activation='softmax')
])

# Available activations: 'relu', 'sigmoid', 'tanh', 'elu', 'selu',
# 'gelu', 'swish', 'mish', 'softmax', 'softplus'

# Custom activation function
@tf.function
def custom_leaky_relu(x, alpha=0.01):
    return tf.where(x > 0, x, alpha * x)

Desirable properties of activation functions

Several mathematical and practical properties influence the effectiveness of an activation function:

Nonlinearity. The core requirement. Without nonlinearity, deep networks collapse to a single linear layer.
Differentiability. Gradient-based optimization (backpropagation) requires computing derivatives. Functions like ReLU that are not differentiable at a single point still work in practice because the non-differentiable point has measure zero.
Non-saturating gradients. Functions whose gradients approach zero for large inputs (like sigmoid and tanh) cause the vanishing gradient problem. Non-saturating functions like ReLU maintain useful gradients across a wider input range.
Computational efficiency. Simpler functions (ReLU requires only a comparison) enable faster training, which matters at scale.
Zero-centered output. Outputs centered around zero (as with tanh) can improve gradient flow and convergence speed compared to outputs that are always positive (as with sigmoid or ReLU).
Bounded or unbounded range. Bounded outputs (sigmoid, tanh) can stabilize training but may limit representational capacity. Unbounded outputs (ReLU, GELU) allow for larger activations but may require careful initialization and normalization.
Monotonicity. Traditional wisdom favored monotonic functions, but recent successful activations (GELU, Swish, Mish) are non-monotonic, suggesting monotonicity is not strictly necessary.

References

Rosenblatt, F. (1958). "The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain." *Psychological Review*, 65(6), 386-408.
Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). "Learning representations by back-propagating errors." *Nature*, 323(6088), 533-536.
Cybenko, G. (1989). "Approximation by superpositions of a sigmoidal function." *Mathematics of Control, Signals and Systems*, 2(4), 303-314.
Nair, V. and Hinton, G. E. (2010). "Rectified Linear Units Improve Restricted Boltzmann Machines." *Proceedings of the 27th International Conference on Machine Learning (ICML)*, 807-814.
Glorot, X., Bordes, A., and Bengio, Y. (2011). "Deep Sparse Rectifier Neural Networks." *Proceedings of the 14th International Conference on Artificial Intelligence and Statistics (AISTATS)*, 315-323.
Maas, A. L., Hannun, A. Y., and Ng, A. Y. (2013). "Rectifier Nonlinearities Improve Neural Network Acoustic Models." *ICML Workshop on Deep Learning for Audio, Speech, and Language Processing*.
He, K., Zhang, X., Ren, S., and Sun, J. (2015). "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification." *Proceedings of the IEEE International Conference on Computer Vision (ICCV)*, 1026-1034.
Clevert, D.-A., Unterthiner, T., and Hochreiter, S. (2016). "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)." *Proceedings of the International Conference on Learning Representations (ICLR)*.
Hendrycks, D. and Gimpel, K. (2016). "Gaussian Error Linear Units (GELUs)." *arXiv preprint arXiv:1606.08415*.
Klambauer, G., Unterthiner, T., Mayr, A., and Hochreiter, S. (2017). "Self-Normalizing Neural Networks." *Advances in Neural Information Processing Systems (NeurIPS)*, 30.
Ramachandran, P., Zoph, B., and Le, Q. V. (2017). "Searching for Activation Functions." *arXiv preprint arXiv:1710.05941*.
Dauphin, Y. N., Fan, A., Auli, M., and Grangier, D. (2017). "Language Modeling with Gated Convolutional Networks." *Proceedings of the 34th International Conference on Machine Learning (ICML)*, 933-941.
Misra, D. (2019). "Mish: A Self Regularized Non-Monotonic Activation Function." *arXiv preprint arXiv:1908.08681*.
Shazeer, N. (2020). "GLU Variants Improve Transformer." *arXiv preprint arXiv:2002.05202*.
Dubey, S. R., Singh, S. K., and Chaudhuri, B. B. (2022). "Activation Functions in Deep Learning: A Comprehensive Survey and Benchmark." *Neurocomputing*, 503, 92-108.

Explain like I'm 5 (ELI5)

Why nonlinearity matters

Historical evolution

Step functions and perceptrons (1950s-1960s)

Sigmoid and tanh era (1980s-2000s)

ReLU revolution (2010-2015)

Modern smooth activations (2016-present)

Common activation functions

Sigmoid (logistic)

Tanh (hyperbolic tangent)

ReLU (Rectified Linear Unit)

Leaky ReLU

Parametric ReLU (PReLU)

ELU (Exponential Linear Unit)

SELU (Scaled Exponential Linear Unit)

GELU (Gaussian Error Linear Unit)

Swish / SiLU (Sigmoid Linear Unit)

Mish

Softmax

Softplus

GLU and SwiGLU (Gated Linear Units)

Comparison table

How to choose an activation function

Hidden layers

Output layers

Architecture-specific conventions

Activation functions in transformers

Implementation examples

PyTorch

TensorFlow / Keras

Desirable properties of activation functions

See also

References

Improve this article

Related Articles

GELU (Gaussian Error Linear Unit)

Multi-head Latent Attention

ARC-AGI 2

LeNet

Bayesian Neural Network

Decoder

Explain like I'm 5 (ELI5)

Why nonlinearity matters

Historical evolution

Step functions and perceptrons (1950s-1960s)

Sigmoid and tanh era (1980s-2000s)

ReLU revolution (2010-2015)

Modern smooth activations (2016-present)

Common activation functions

Sigmoid (logistic)

Tanh (hyperbolic tangent)

ReLU (Rectified Linear Unit)

Leaky ReLU

Parametric ReLU (PReLU)

ELU (Exponential Linear Unit)

SELU (Scaled Exponential Linear Unit)

GELU (Gaussian Error Linear Unit)

Swish / SiLU (Sigmoid Linear Unit)

Mish

Softmax

Softplus

GLU and SwiGLU (Gated Linear Units)

Comparison table

How to choose an activation function

Hidden layers

Output layers

Architecture-specific conventions

Activation functions in transformers

Implementation examples

PyTorch

TensorFlow / Keras

Desirable properties of activation functions

See also

References

Related Articles

GELU (Gaussian Error Linear Unit)

Multi-head Latent Attention

ARC-AGI 2

LeNet

Bayesian Neural Network