# GELU (Gaussian Error Linear Unit)

> Source: https://aiwiki.ai/wiki/gelu
> Updated: 2026-06-22
> Fact-checked: 2026-11-05
> Categories: Artificial Intelligence, Deep Learning, Neural Networks
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "GELU (Gaussian Error Linear Unit)." aiwiki.ai, 22 Jun 2026. https://aiwiki.ai/wiki/gelu
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

The **Gaussian Error Linear Unit (GELU)** is a smooth, non-monotonic [activation function](https://aiwiki.ai/wiki/activation_function) defined as `GELU(x) = x · Φ(x)`, where `Φ(x)` is the cumulative distribution function of the standard [normal distribution](https://aiwiki.ai/wiki/normal_distribution). It was introduced by [Dan Hendrycks](https://aiwiki.ai/wiki/dan_hendrycks) and [Kevin Gimpel](https://aiwiki.ai/wiki/kevin_gimpel) in the 2016 paper *Gaussian Error Linear Units (GELUs)*, posted to [arXiv](https://aiwiki.ai/wiki/arxiv) on 27 June 2016 as 1606.08415, and it is the default nonlinearity in the feed-forward layers of [BERT](https://aiwiki.ai/wiki/bert), the GPT series, and most [transformer](https://aiwiki.ai/wiki/transformers) models built between 2018 and 2022.[1] The paper's defining sentence is that the GELU "weights inputs by their value, rather than gates inputs by their sign as in ReLUs."[1]

Hendrycks was then at the University of Chicago (later UC Berkeley), and Gimpel was at the [Toyota Technological Institute at Chicago](https://aiwiki.ai/wiki/ttic). The earliest preprint was titled *Bridging Nonlinearities and Stochastic Regularizers with Gaussian Error Linear Units*, and the authors kept revising the manuscript through June 2023 (final revision v5).[1]

BERT, [GPT-2](https://aiwiki.ai/wiki/gpt-2), [GPT-3](https://aiwiki.ai/wiki/gpt-3), [RoBERTa](https://aiwiki.ai/wiki/roberta), [ALBERT](https://aiwiki.ai/wiki/albert), [ELECTRA](https://aiwiki.ai/wiki/electra), the original GPT (often called [GPT-1](https://aiwiki.ai/wiki/gpt-1)), and the original [Vision Transformer](https://aiwiki.ai/wiki/vision_transformer) all use GELU between the two linear layers of each [feed-forward](https://aiwiki.ai/wiki/feedforward_neural_network_ffn) block. [OpenAI](https://aiwiki.ai/wiki/openai)'s [Whisper](https://aiwiki.ai/wiki/whisper) speech-recognition model uses it as well.[11] Newer architectures like [PaLM](https://aiwiki.ai/wiki/palm) and [LLaMA](https://aiwiki.ai/wiki/llama) have moved to gated variants such as [SwiGLU](https://aiwiki.ai/wiki/swiglu), but GELU remains common in encoder models, vision transformers, and codebases descended from BERT.

## Quick facts

| | |
|---|---|
| Introduced | June 2016 |
| Paper | *Gaussian Error Linear Units (GELUs)* |
| arXiv ID | 1606.08415 (versions v1 through v5) |
| Authors | [Dan Hendrycks](https://aiwiki.ai/wiki/dan_hendrycks), Kevin Gimpel |
| Affiliations | University of Chicago / UC Berkeley; Toyota Technological Institute at Chicago |
| Exact formula | `GELU(x) = x · Φ(x) = (x / 2) · (1 + erf(x / √2))` |
| Tanh approximation | `0.5 · x · (1 + tanh(√(2/π) · (x + 0.044715 · x³)))` |
| Sigmoid approximation | `x · σ(1.702 · x)` |
| Smooth | Yes (infinitely differentiable) |
| Monotonic | No (small dip near `x ≈ -0.75`) |
| Used in | [BERT](https://aiwiki.ai/wiki/bert), [GPT-1](https://aiwiki.ai/wiki/gpt-1), [GPT-2](https://aiwiki.ai/wiki/gpt-2), [GPT-3](https://aiwiki.ai/wiki/gpt-3), [RoBERTa](https://aiwiki.ai/wiki/roberta), ALBERT, ELECTRA, T5 v1.1 (via GeGLU), [Vision Transformer](https://aiwiki.ai/wiki/vision_transformer), Whisper |
| Frameworks | [PyTorch](https://aiwiki.ai/wiki/pytorch), [TensorFlow](https://aiwiki.ai/wiki/tensorflow), [JAX](https://aiwiki.ai/wiki/jax), [Hugging Face Transformers](https://aiwiki.ai/wiki/transformers_library) |

<figure class="embedvideo">
<iframe src="https://www.youtube-nocookie.com/embed/2FaI2Fen1mQ" title="The GELU, SiLU and SwiGLU activation functions, clearly explained!!!"></iframe>
<figcaption>"The GELU, SiLU and SwiGLU activation functions, clearly explained!!!", video from StatQuest with Josh Starmer, published September 8, 2026. <a href="https://www.youtube.com/watch?v=2FaI2Fen1mQ">Watch on YouTube</a>.</figcaption>
</figure>

## What problem does GELU solve?

Before GELU, the dominant nonlinearity in deep networks was the [rectified linear unit](https://aiwiki.ai/wiki/relu) (ReLU), introduced in its modern form by Nair and Hinton (2010) and popularised by [AlexNet](https://aiwiki.ai/wiki/alexnet) (Krizhevsky, Sutskever, Hinton, 2012). ReLU is `max(0, x)`. It is cheap, [sparsity](https://aiwiki.ai/wiki/sparsity)-inducing, and largely solves the vanishing-gradient problem that plagued sigmoidal networks. It also has a well-known failure mode usually called the *dead ReLU* problem: once a unit has a sufficiently negative pre-activation, its gradient is exactly zero, and standard [gradient descent](https://aiwiki.ai/wiki/gradient_descent) cannot recover it.

A family of patches followed. **Leaky ReLU** (Maas, Hannun, Ng, 2013) replaces the flat negative half with a small linear slope `αx`.[16] **PReLU** (He et al., 2015) makes that slope a learned parameter.[17] **ELU** (Clevert, Unterthiner, Hochreiter, 2015) replaces the negative half with `α · (eˣ - 1)`, giving a smooth saturating curve below zero.[14] **SELU** (Klambauer et al., 2017) adds a fixed multiplicative scale chosen so successive layers self-normalise their activation statistics.[15]

All of these are deterministic point-wise functions of the pre-activation. In parallel, regularisation work was producing increasingly sophisticated stochastic perturbations of the forward pass: [dropout](https://aiwiki.ai/wiki/dropout_regularization) (Srivastava et al., 2014), zoneout (Krueger et al., 2016), and stochastic depth (Huang et al., 2016). Hendrycks and Gimpel's contribution was to notice that a particular kind of stochastic gating gives rise, in expectation, to a smooth deterministic activation that outperforms ReLU and ELU.[1] The paper contrasts the two by how they treat the input: an activation function can be read as a way to weight or gate its input by some function of itself. ReLU weights the input by `1[x > 0]`, ELU weights the negative half by `(eˣ - 1) / x`, and GELU weights it by the probability that a Gaussian random variable is below the input.[1]

## How is GELU defined?

The exact definition is

```
GELU(x) = x · Φ(x)
```

where `Φ` is the standard normal CDF. Equivalently, using the [error function](https://aiwiki.ai/wiki/error_function) `erf`:

```
GELU(x) = 0.5 · x · (1 + erf(x / √2))
```

The function is smooth (infinitely differentiable), non-monotonic, and approximately linear for large positive `x`. For negative `x` it stays small but never identically zero, so unlike ReLU it does not produce dead units that always output zero. Asymptotically, GELU coincides with ReLU: `GELU(x) → x` as `x → +∞` and `GELU(x) → 0` as `x → -∞`.

A few quick numerical values give a feel for the shape:

| Input `x` | `Φ(x)` | `GELU(x)` |
|---|---|---|
| -3 | 0.00135 | -0.00405 |
| -2 | 0.0228 | -0.0455 |
| -1 | 0.1587 | -0.1587 |
| -0.75 | 0.2266 | -0.170 |
| -0.5 | 0.3085 | -0.1543 |
| 0 | 0.5 | 0 |
| 0.5 | 0.6915 | 0.3457 |
| 1 | 0.8413 | 0.8413 |
| 2 | 0.9772 | 1.9545 |
| 3 | 0.99865 | 2.996 |

GELU has a small negative dip near `x ≈ -0.75`, where it bottoms out at roughly -0.170 (the global minimum). Past `x ≈ 2` it is essentially equal to `x`. Past `x ≈ -3` it is essentially zero. Figure 1 of the original paper plots GELU alongside ReLU and ELU and is the reference picture most readers carry in their heads.[1]

The derivative of the exact form is

```
GELU'(x) = Φ(x) + x · φ(x)
```

where `φ(x) = (1 / √(2π)) · exp(-x² / 2)` is the standard normal probability density function. The autograd engines in [PyTorch](https://aiwiki.ai/wiki/pytorch), [TensorFlow](https://aiwiki.ai/wiki/tensorflow), and [JAX](https://aiwiki.ai/wiki/jax) all generate this analytically. The derivative is bounded (its supremum is about 1.13, at `x = √2 ≈ 1.41`), so GELU does not amplify gradients in the way some unbounded activations can.

## Why a Gaussian gate? The stochastic regularizer interpretation

The original paper motivates GELU as a deterministic version of an input-dependent stochastic regularizer.[1] Suppose you multiply each input by a Bernoulli mask `m`, where `m = 1` with probability `Φ(x)` and `m = 0` otherwise. The expected value of the output is

```
E[m · x] = Φ(x) · x = GELU(x)
```

Under this view, GELU is the expected value of an input that is either kept (probability `Φ(x)`) or dropped (probability `1 - Φ(x)`), where the keep probability rises with the input's magnitude on the positive side. Hendrycks and Gimpel describe it as combining the gating behaviour of ReLU and the stochasticity of [dropout](https://aiwiki.ai/wiki/dropout) and zoneout into a single deterministic nonlinearity.[1] Inputs that look more like noise (small in absolute value, especially negative) get attenuated more aggressively in expectation than inputs that are clearly signal.

This framing explains the slight negative bump: when `x` is moderately negative, the mask is unlikely to be 1, but if it is, the output is small and negative. The expected value reflects that. The paper also reports that networks trained solely with the stochastic mask `m ~ Bernoulli(Φ(x))`, with no other nonlinearity, were competitive on MNIST and TIMIT.[1] In practice the deterministic version is what everyone uses.

The choice of the Gaussian CDF rather than the logistic sigmoid matches the assumption that pre-activations in well-initialised deep networks are approximately Gaussian. The sigmoid approximation `x · σ(1.702 · x)` substitutes the logistic CDF, rescaled so that `σ(1.702 · x)` closely tracks `Φ(x)`.

## What are the GELU approximations?

The exact form requires evaluating `erf`, which is more expensive than the elementary operations used by ReLU or the [sigmoid function](https://aiwiki.ai/wiki/sigmoid_function). The 2016 paper proposed two approximations that became standard practice for years.[1]

**Tanh approximation:**

```
GELU(x) ≈ 0.5 · x · (1 + tanh(√(2/π) · (x + 0.044715 · x³)))
```

The constant `√(2/π) ≈ 0.7978845608` and the cubic coefficient `0.044715` come from an approximation to the normal CDF that the paper credits to Choudhury (2014).[1] This is the form used in the GPT-1 and GPT-2 reference code and in Google's BERT repository from February 2019 (the October 2018 BERT release used the exact `erf` form), and a great deal of pretrained-checkpoint code still uses it.[24][25]

**Sigmoid approximation:**

```
GELU(x) ≈ x · σ(1.702 · x)
```

where `σ` is the logistic sigmoid. This is faster than the tanh approximation but less accurate. It is rarely used in production but appears in the original paper as a convenient closed form.[1] The constant `1.702` is the scaling at which the logistic CDF `σ(1.702 · x)` closely tracks the Gaussian CDF `Φ(x)`, and it is also the factor that turns plain SiLU (`x · σ(x)`) into a GELU approximation.[1][13]

**Exact form** uses `erf` directly. Most modern hardware has fast `erf` instructions or libm implementations, so on contemporary GPUs the exact form is competitive with the tanh approximation in wall-clock time. PyTorch exposes the choice through `torch.nn.functional.gelu(x, approximate='none')` (the default, which calls `erf`) and `approximate='tanh'` (the formulation used in OpenAI's GPT code).[20] TensorFlow exposes the same selection through `tf.nn.gelu` with an `approximate=True/False` flag.[21] JAX provides `jax.nn.gelu` with an `approximate` keyword argument that defaults to `True`.[22]

The maximum absolute deviation between the tanh approximation and the exact `erf` form is on the order of `1e-4` over the range `[-5, 5]`. The sigmoid approximation deviates by closer to `1e-2` in the same range.

## Properties

- **Smooth.** Continuous derivatives of all orders, unlike ReLU (kink at zero) or Leaky ReLU.
- **Non-monotonic.** Decreases on `(-∞, -0.75)` then increases, so it is not one-to-one near the origin.
- **Differentiable everywhere.** No special handling for `x = 0`.
- **Asymptotically ReLU.** `GELU(x) - max(0, x) → 0` as `|x| → ∞`.
- **Bounded below.** `GELU(x) ≥ -0.170` for all real `x`.
- **Approximately zero-centred outputs.** Easier to layer-normalise than ReLU.
- **Self-gated.** The function gates each input by some monotone function of itself, putting it in the same family as Swish/SiLU and Mish.

GELU's small negative bump means that two distinct inputs can produce the same output, which complicates some theoretical analyses but does not seem to hurt training in practice.

## How does GELU differ from ReLU and other activations?

| Activation | Formula | Smooth | Non-monotonic | Notes |
|---|---|---|---|---|
| [ReLU](https://aiwiki.ai/wiki/relu) | `max(0, x)` | No | No | Cheapest. Standard before transformers. Has dead-unit problem. |
| Leaky ReLU | `max(αx, x)`, `α = 0.01` typical | No | No | Avoids dead units. Cheap. |
| PReLU | `max(αx, x)`, `α` learned | No | No | Adds parameters per channel. |
| [ELU](https://aiwiki.ai/wiki/elu) | `x` if `x > 0`, else `α(eˣ - 1)` | Yes | No | Smooth saturating negatives. |
| SELU | Scaled ELU with fixed `α`, `λ` | Yes | No | Self-normalising. |
| [SiLU / Swish](https://aiwiki.ai/wiki/activation_function) | `x · σ(x)` | Yes | Yes | GELU's sigmoid form with the `1.702` coefficient set to 1. |
| GELU (exact) | `x · Φ(x)` | Yes | Yes | Gaussian gate. Min around -0.170. |
| GELU (tanh approx) | `0.5 · x · (1 + tanh(√(2/π) · (x + 0.044715 · x³)))` | Yes | Yes | [BERT](https://aiwiki.ai/wiki/bert)/[GPT-2](https://aiwiki.ai/wiki/gpt-2) default. |
| [Mish](https://aiwiki.ai/wiki/mish) | `x · tanh(softplus(x))` | Yes | Yes | Min about -0.31. |

The key difference from ReLU is captured by the original abstract: GELU "weights inputs by their value, rather than gates inputs by their sign."[1] ReLU makes a hard binary decision based on the sign of the pre-activation; GELU applies a smooth, probabilistic weighting that keeps a small nonzero signal for negative inputs and so avoids dead units.

The SiLU function `x · σ(x)` was introduced and named in the same 2016 paper, as a separate activation from the GELU sigmoid approximation (which uses the 1.702 coefficient), and was rediscovered by Ramachandran, Zoph, and Le of Google Brain in 2017 under the name Swish.[1][13] Libraries such as TensorFlow and PyTorch later adopted the name SiLU. The two names refer to the same function. GELU is sometimes described as Swish with a Gaussian gate instead of a sigmoid one. In practice GELU and SiLU produce very similar shapes, with GELU having a slightly more pronounced negative dip.

## What did the original experiments show?

Hendrycks and Gimpel ran experiments on [MNIST](https://aiwiki.ai/wiki/mnist) classification, MNIST autoencoding, Twitter part-of-speech tagging, TIMIT frame classification, [CIFAR-10](https://aiwiki.ai/wiki/cifar_10), and CIFAR-100, reporting "performance improvements across all considered computer vision, natural language processing, and speech tasks."[1] GELU had the best median result against ReLU and ELU in every setting they tested, with the largest gap on CIFAR-100. The differences were modest but consistent.[1]

- **MNIST classification.** Eight-layer, 128-neuron-wide fully connected networks trained for 50 epochs with a [batch size](https://aiwiki.ai/wiki/batch_size) of 128. GELU tended to have the lowest median training log loss with and without dropout, and was at least as robust as ReLU and ELU to noised inputs.
- **MNIST autoencoding.** Deep [autoencoder](https://aiwiki.ai/wiki/autoencoder) with layers of width 1000, 500, 250, 30, 250, 500, 1000. GELU achieved lower reconstruction error than ReLU or ELU at both learning rates tested.
- **Twitter POS tagging.** Two-layer network on part-of-speech-annotated tweets. Median test error 12.57% for GELU, 12.67% for ReLU, and 12.91% for ELU.
- **CIFAR-10 and CIFAR-100.** A 9-layer CNN trained for 200 epochs on CIFAR-10 (median error 7.89% for GELU, 8.16% for ReLU, 8.41% for ELU) and a 40-layer Wide ResNet with widening factor 4 trained for 50 epochs on CIFAR-100 (20.74%, 21.77%, and 22.98%).
- **TIMIT frame classification.** Five-layer, 2048-neuron-wide fully connected classifier. Median test error 29.3% for GELU versus 29.5% for ReLU and 29.6% for ELU.

These benchmarks predate the transformer era. The reason GELU caught on was not the MNIST result but the fact that the GPT and BERT teams adopted it for the FFN layers, and pretty much every transformer paper after that used whatever the BERT codebase used.

## Which transformer models use GELU?

GELU was not the first nonlinearity used inside a transformer. The original *[Attention Is All You Need](https://aiwiki.ai/wiki/attention_is_all_you_need)* paper (Vaswani et al., 2017) used ReLU in the position-wise feed-forward sublayer.[19] The shift to GELU happened with the next generation of pretrained models.

| Model | Year | FFN activation | Approximation in reference code |
|---|---|---|---|
| Original Transformer | 2017 | ReLU | n/a |
| GPT ([GPT-1](https://aiwiki.ai/wiki/gpt-1)) | 2018 | GELU | Tanh approximation |
| [BERT](https://aiwiki.ai/wiki/bert) | 2018 | GELU | Exact (erf) at the October 2018 release; tanh in the repository from February 2019[24] |
| [GPT-2](https://aiwiki.ai/wiki/gpt-2) | 2019 | GELU | Tanh approximation |
| [RoBERTa](https://aiwiki.ai/wiki/roberta) | 2019 | GELU | Exact (fairseq `gelu`)[27] |
| XLNet | 2019 | GELU | Tanh approximation |
| [ALBERT](https://aiwiki.ai/wiki/albert) | 2019 | GELU | Tanh approximation |
| T5 (v1.0) | 2019 | ReLU | n/a |
| T5 (v1.1) | 2020 | GeGLU (uses GELU) | Tanh approximation[28] |
| [ELECTRA](https://aiwiki.ai/wiki/electra) | 2020 | GELU | Tanh approximation |
| [GPT-3](https://aiwiki.ai/wiki/gpt-3) | 2020 | GELU | No public code (architecture follows GPT-2) |
| [Vision Transformer](https://aiwiki.ai/wiki/vision_transformer) | 2020 | GELU | Tanh approximation |
| [Whisper](https://aiwiki.ai/wiki/whisper) | 2022 | GELU | Exact (PyTorch default)[26] |
| [PaLM](https://aiwiki.ai/wiki/palm) | 2022 | [SwiGLU](https://aiwiki.ai/wiki/swiglu) | n/a |
| [LLaMA](https://aiwiki.ai/wiki/llama) | 2023 | [SwiGLU](https://aiwiki.ai/wiki/swiglu) | n/a |

BERT (Devlin et al., October 2018) is the single largest cause of GELU's spread.[2] Its public TensorFlow reference implementation first defined `gelu` with the exact `erf` form and switched to the tanh approximation in February 2019, and much of the code that built on BERT inherited one form or the other.[24] The OpenAI GPT-1 paper (Radford et al., June 2018) also used GELU, but BERT's open-source release was the catalyst that pushed it into nearly every transformer codebase between 2018 and 2022.[3]

## How does GELU compare to SwiGLU?

The activation question got more complicated in February 2020 with [Noam Shazeer](https://aiwiki.ai/wiki/noam_shazeer)'s paper *GLU Variants Improve Transformer* (arXiv:2002.05202), which proposed using gated linear units in place of the standard FFN.[12] Two variants became popular:

- **GeGLU**: `GeGLU(x, W, V, b, c) = GELU(xW + b) ⊗ (xV + c)`. T5 v1.1 and the [LaMDA](https://aiwiki.ai/wiki/lamda) family use this.[12]
- **SwiGLU**: same structure but with SiLU/Swish in place of GELU.[12] PaLM and LLaMA adopted this, and the Mistral and Mixtral lines kept it.

Gated FFNs use three weight matrices instead of two and reduce the inner dimension by `2/3` to keep parameter count roughly constant.[12] They consistently produce small but reliable perplexity improvements over plain GELU FFNs. Most large open-weight models released after 2022 use SwiGLU. GELU is still the default in encoder-only and encoder-decoder models, in vision transformers, and in any transformer codebase matching the BERT lineage. Shazeer offers no theoretical account of the gains, concluding: "We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence."[12]

## Implementation

A reference PyTorch implementation of all three forms looks like this:

```python
import math
import torch
import torch.nn.functional as F

def gelu_exact(x):
    return 0.5 * x * (1.0 + torch.erf(x / math.sqrt(2.0)))

def gelu_tanh(x):
    c = math.sqrt(2.0 / math.pi)
    return 0.5 * x * (1.0 + torch.tanh(c * (x + 0.044715 * x.pow(3))))

def gelu_sigmoid(x):
    return x * torch.sigmoid(1.702 * x)

# Library-provided versions
y1 = F.gelu(x)                        # exact, uses erf
y2 = F.gelu(x, approximate='tanh')    # tanh approximation
```

The library version is preferable in real code because it dispatches to fused kernels on supported hardware.

### Framework APIs

- **PyTorch.** `torch.nn.GELU(approximate='none' | 'tanh')` and `torch.nn.functional.gelu`. The `approximate='tanh'` flag was added in PyTorch 1.12 (June 2022); before that, only the exact form was available.[20]
- **TensorFlow.** `tf.keras.activations.gelu(x, approximate=False)`. The `approximate` flag has been present since TensorFlow 2.4.[21]
- **JAX.** `jax.nn.gelu(x, approximate=True)`. The default is the tanh approximation, the opposite of PyTorch's default.[22]
- **[Hugging Face](https://aiwiki.ai/wiki/hugging_face) Transformers.** The `transformers.activations` module exposes `gelu` (exact `erf`), `gelu_new` (BERT tanh approximation, kept for checkpoint compatibility), `gelu_fast` (cheaper tanh variant), `gelu_pytorch_tanh` (calls `F.gelu(x, approximate='tanh')`), `quick_gelu` (sigmoid approximation, used by CLIP), and `gelu_python` (a pure-Python reference). The names are historical artefacts.[23]

The `quick_gelu` variant matches the `QuickGELU` module in OpenAI's CLIP code, which is used in the transformer blocks of both its text encoder and its ViT image encoder.[29] Loading CLIP weights with the wrong activation produces clearly degraded zero-shot accuracy, which is a recurring source of confusion.

## Numerical reproducibility pitfalls

Small numerical differences between the exact and tanh approximations can affect reproducibility. Hugging Face Transformers keeps both forms for this reason: its `gelu` is documented as the original implementation in Google's BERT repository when it was first created (exact `erf`), `gelu_new` as the version later used in that repository and identical to OpenAI GPT (tanh), and `gelu_pytorch_tanh` calls PyTorch's native `approximate='tanh'` kernel, which the library notes is not an exact numerical match to `gelu_new` because of rounding.[23][24]

Practical rules:

- **Match the activation used at training time.** If a checkpoint was trained with the tanh approximation, evaluate it with the tanh approximation.
- **For new training runs, the exact `erf` form is the safer default.** It is framework-agnostic and avoids the historical baggage of competing approximations.
- **Test for numerical equivalence carefully.** Comparing forward passes between frameworks at a tolerance of `1e-6` will fail if one side uses the exact form and the other uses the tanh approximation; `1e-3` is more realistic.

A related issue affects half-precision (FP16/BF16) training. The tanh approximation involves a cubic term `0.044715 · x³`, which can overflow in FP16 for large `x` even though the final output is well-behaved. Most framework implementations compute the cubic in higher precision to avoid this. The exact `erf` form is more numerically stable in low precision.

## Computational considerations

On modern GPUs, the cost of `erf` is dominated by the surrounding `elementwise * matmul + bias` pattern, so the activation choice is not the bottleneck for most transformer training. The tanh approximation can be marginally faster on hardware with weak transcendental support, which is one reason older code paths still use it.

The more practical reason to keep using the tanh form is checkpoint compatibility. If you are running a pretrained model whose weights were tuned against the tanh approximation, switching to the exact form changes the function values slightly, and the model's downstream metrics will drift. The drift is small but reproducible. Derivatives are inexpensive in either form: the exact derivative is `Φ(x) + x · φ(x)`, and every autograd framework computes the tanh derivative correctly.

## See also

- [Activation Function](https://aiwiki.ai/wiki/activation_function)
- [ReLU](https://aiwiki.ai/wiki/relu)
- [SiLU / Swish](https://aiwiki.ai/wiki/silu)
- [SwiGLU](https://aiwiki.ai/wiki/swiglu)
- [GLU](https://aiwiki.ai/wiki/glu)
- [Transformer](https://aiwiki.ai/wiki/transformer)
- [BERT](https://aiwiki.ai/wiki/bert)

## References

1. Hendrycks, Dan; Gimpel, Kevin. *Gaussian Error Linear Units (GELUs)*. arXiv:1606.08415, 27 June 2016 (final revision v5, June 2023). https://arxiv.org/abs/1606.08415
2. Devlin, Jacob; Chang, Ming-Wei; Lee, Kenton; Toutanova, Kristina. *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*. arXiv:1810.04805, October 2018.
3. Radford, Alec; Narasimhan, Karthik; Salimans, Tim; Sutskever, Ilya. *Improving Language Understanding by Generative Pre-Training* (GPT-1), OpenAI, June 2018.
4. Radford, Alec; Wu, Jeffrey; Child, Rewon; Luan, David; Amodei, Dario; Sutskever, Ilya. *Language Models are Unsupervised Multitask Learners* (GPT-2), OpenAI, 2019.
5. Brown, Tom et al. *Language Models are Few-Shot Learners* (GPT-3). arXiv:2005.14165, May 2020.
6. Liu, Yinhan et al. *RoBERTa: A Robustly Optimized BERT Pretraining Approach*. arXiv:1907.11692, July 2019.
7. Lan, Zhenzhong et al. *ALBERT: A Lite BERT for Self-supervised Learning of Language Representations*. arXiv:1909.11942, September 2019.
8. Clark, Kevin; Luong, Minh-Thang; Le, Quoc V.; Manning, Christopher D. *ELECTRA*. arXiv:2003.10555, March 2020.
9. Raffel, Colin et al. *Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer* (T5). arXiv:1910.10683, October 2019.
10. Dosovitskiy, Alexey et al. *An Image is Worth 16x16 Words* (Vision Transformer). arXiv:2010.11929, October 2020.
11. Radford, Alec et al. *Robust Speech Recognition via Large-Scale Weak Supervision* (Whisper). arXiv:2212.04356, December 2022.
12. Shazeer, Noam. *GLU Variants Improve Transformer*. arXiv:2002.05202, February 2020. https://arxiv.org/abs/2002.05202
13. Ramachandran, Prajit; Zoph, Barret; Le, Quoc V. *Searching for Activation Functions* (Swish). arXiv:1710.05941, October 2017.
14. Clevert, Djork-Arne; Unterthiner, Thomas; Hochreiter, Sepp. *Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)*. arXiv:1511.07289, November 2015.
15. Klambauer, Gunter et al. *Self-Normalizing Neural Networks* (SELU). arXiv:1706.02515, June 2017.
16. Maas, Andrew L.; Hannun, Awni Y.; Ng, Andrew Y. *Rectifier Nonlinearities Improve Neural Network Acoustic Models* (Leaky ReLU). ICML 2013.
17. He, Kaiming et al. *Delving Deep into Rectifiers* (PReLU). arXiv:1502.01852, February 2015.
18. Misra, Diganta. *Mish: A Self Regularized Non-Monotonic Activation Function*. arXiv:1908.08681, August 2019.
19. Vaswani, Ashish et al. *Attention Is All You Need*. arXiv:1706.03762, June 2017.
20. PyTorch documentation, `torch.nn.functional.gelu`. https://pytorch.org/docs/stable/generated/torch.nn.functional.gelu.html
21. TensorFlow documentation, `tf.keras.activations.gelu`. https://www.tensorflow.org/api_docs/python/tf/keras/activations/gelu
22. JAX documentation, `jax.nn.gelu`. https://jax.readthedocs.io/en/latest/_autosummary/jax.nn.gelu.html
23. Hugging Face Transformers, `transformers.activations` source. https://github.com/huggingface/transformers/blob/main/src/transformers/activations.py
24. Google Research, `bert` repository, `modeling.py` (initial release of 31 October 2018 uses `tf.erf`; tanh form from the commit of 7 February 2019). https://github.com/google-research/bert/blob/master/modeling.py
25. OpenAI, `gpt-2` repository, `src/model.py`. https://github.com/openai/gpt-2/blob/master/src/model.py
26. OpenAI, `whisper` repository, `whisper/model.py`. https://github.com/openai/whisper/blob/main/whisper/model.py
27. fairseq, `fairseq/modules/gelu.py`. https://github.com/facebookresearch/fairseq/blob/main/fairseq/modules/gelu.py
28. Hugging Face Transformers, `configuration_t5.py` (maps T5 v1.1's `gated-gelu` to `gelu_new`). https://github.com/huggingface/transformers/blob/main/src/transformers/models/t5/configuration_t5.py
29. OpenAI, `CLIP` repository, `clip/model.py`. https://github.com/openai/CLIP/blob/main/clip/model.py

