Softmax

Softmax is a function that converts a finite vector of real-valued scores into a vector of positive numbers that sum to one. If the scores are logits for mutually exclusive outcomes, the result can parameterize a categorical probability distribution. Softmax is widely used as an activation function in multi-class classification, as the normalization step in attention, and as an action-selection rule. It is also called the normalized exponential function.^[1][2]

Softmax preserves score ordering and turns score differences into probability ratios. It does not, by itself, show that the resulting numbers are calibrated probabilities or that the model is correct. Those interpretations depend on the statistical model, training objective, data, and evaluation procedure. The computation must also be implemented carefully: direct exponentiation can overflow, while very negative terms can underflow in floating-point arithmetic.^[3]

Definition

For a vector of scores

$$\mathbf{z}=(z_1,\ldots,z_K)\in\mathbb{R}^{K}$$

and a positive temperature T, softmax is

$$\operatorname{softmax}_T(\mathbf{z})_i = \frac{\exp(z_i/T)} {\sum_{j=1}^{K}\exp(z_j/T)}, \qquad i=1,\ldots,K.$$

The ordinary softmax uses T = 1. Some mathematical treatments instead use an inverse-temperature parameter lambda = 1/T and write exp(lambda z_i).^[2]

For any finite real inputs and K >= 2, every output satisfies

$$0 < \operatorname{softmax}_T(\mathbf{z})_i < 1, \qquad \sum_{i=1}^{K}\operatorname{softmax}_T(\mathbf{z})_i=1.$$

The output is therefore in the interior of the (K - 1)-dimensional probability simplex. The strict inequalities require finite inputs. Implementations often use negative infinity as a mask, in which case masked positions are assigned zero by convention.^[4][21] NaN and positive-infinity inputs need separate handling because ordinary real-number identities do not determine all floating-point results.

Ratios and ordering

For any two components,

$$\frac{p_i}{p_j} = \exp\left(\frac{z_i-z_j}{T}\right).$$

This identity shows three important facts:

1. Only score differences affect the output.
2. If z_i > z_j, then p_i > p_j.
3. A difference of T log r produces probability odds p_i/p_j = r.

The largest score therefore has the largest softmax component, so softmax preserves the set of maximizing indices. It changes a score vector into a smooth distribution without changing which class wins under argmax.

Shift invariance

Adding the same finite constant c to every score has no effect:

$$\operatorname{softmax}_T(\mathbf{z}+c\mathbf{1}) = \operatorname{softmax}_T(\mathbf{z}).$$

The common factor exp(c/T) cancels between numerator and denominator. This invariance means that logits are identifiable only up to an additive constant. A K-class distribution with strictly positive probabilities has many logit representations; one is z_i = T log p_i, and adding any shared constant gives another.

Temperature

Temperature rescales every score difference. For a fixed nonconstant vector:

• T > 1 reduces the magnitudes of score differences and produces a distribution closer to uniform.
• 0 < T < 1 amplifies score differences and concentrates mass on the largest components.
• As T approaches infinity, the distribution approaches the uniform vector (1/K,\ldots,1/K).
• As T approaches zero from above, probability mass concentrates on the maximizing components.

The final limit needs a tie qualification. If exactly m components share the maximum, each receives limiting probability 1/m, while every lower component approaches zero. The limit is one-hot only when the maximum is unique.

Temperature changes the probabilities but not the maximizing indices when T is positive. A negative value is not the usual temperature convention and would reverse the ordering. T = 0 is undefined in the formula.

Worked example

Consider three logits

$$\mathbf{z}=(2,1,0).$$

At unit temperature, direct exponentiation gives

$$\exp(\mathbf{z})=(e^2,e,1) \approx(7.389,2.718,1).$$

The normalizer is approximately 11.107, so

$$\operatorname{softmax}(\mathbf{z}) \approx (0.6652,0.2447,0.0900).$$

Subtracting the maximum first gives the equivalent calculation

$$\operatorname{softmax}(2,1,0) = \operatorname{softmax}(0,-1,-2).$$

This second form produces the same real-number result but avoids computing e^2. The odds of the first outcome over the second are e, and the odds of the first over the third are e^2, exactly as the ratio identity predicts.

At T = 2, the scaled logits are (1,0.5,0) and the result is approximately (0.5065,0.3072,0.1863). At T = 0.5, the scaled logits are (4,2,0) and the result is approximately (0.8668,0.1173,0.0159). These examples illustrate flattening and sharpening without changing the top-scoring class.

Log-sum-exp and differential properties

Define the temperature-scaled log-sum-exp function

$$\operatorname{LSE}_T(\mathbf{z}) = T\log\left(\sum_{j=1}^{K}\exp(z_j/T)\right).$$

Its gradient is softmax:

$$\frac{\partial\operatorname{LSE}_T(\mathbf{z})}{\partial z_i} = \operatorname{softmax}_T(\mathbf{z})_i.$$

This connects softmax to convex analysis. Log-sum-exp is convex, and softmax is its monotone gradient map.^[2] It also explains why log-sum-exp and softmax usually appear together in likelihoods and stable software implementations.

Jacobian

Let p = softmax_T(z). The derivative of component p_i with respect to score z_j is

$$\frac{\partial p_i}{\partial z_j} = \frac{1}{T}p_i(\delta_{ij}-p_j),$$

where delta_ij is one when i = j and zero otherwise. In matrix form,

$$J(\mathbf{z}) = \frac{1}{T} \left( \operatorname{diag}(\mathbf{p})-\mathbf{p}\mathbf{p}^{\mathsf T} \right).$$

The Jacobian is symmetric and positive semidefinite. Its rows and columns sum to zero, and the all-ones vector is in its null space. That null direction is the differential form of shift invariance. The off-diagonal entries are nonpositive: increasing one score increases its own probability and decreases every other probability, with all changes summing to zero.^[2]

For a probability vector p, the matrix diag(p) - pp^T is also the covariance matrix of a one-hot categorical random variable. This gives a statistical interpretation to the curvature of log-sum-exp.

Inverse image and boundary

For a fixed positive temperature, softmax maps score vectors onto exactly the interior of the probability simplex. Given any distribution with p_i > 0 for every component, the logits

$$z_i=T\log p_i$$

produce that distribution. Adding a shared constant produces the same result. Conversely, if two finite vectors have the same softmax output, their pairwise score differences are equal, so the vectors differ only by a shared constant. Softmax is therefore one-to-one only after removing that redundant direction, for example by requiring the logits to sum to zero or by fixing one reference logit.

Finite logits cannot represent a simplex boundary point with an exact zero component. A boundary distribution can be approached by sending one or more relative logits toward negative infinity, or represented in an extended-real masked computation. This distinction matters when a model must make structurally impossible outcomes exactly impossible. A dense softmax can make their probabilities arbitrarily small, but only a mask or a sparse alternative makes them exactly zero in the mathematical output.

The image also explains why normalized probabilities do not determine an absolute score scale. The distribution determines log-odds log(p_i/p_j), which recover differences z_i-z_j after multiplication by temperature. It contains no information about a shared offset. If the temperature is also unknown, multiplying both the logits and temperature by the same positive constant leaves the output unchanged, adding another parameterization ambiguity.

Entropy-regularized maximization

Softmax can be characterized as the optimizer of a score-plus-entropy problem:

$$\operatorname{softmax}_T(\mathbf{z}) = \underset{\mathbf{p}\in\Delta^{K-1}}{\operatorname{argmax}} \left\{ \mathbf{p}^{\mathsf T}\mathbf{z} + T H(\mathbf{p}) \right\},$$

where

$$H(\mathbf{p})=-\sum_i p_i\log p_i$$

is Shannon entropy and Delta^(K-1) is the probability simplex. The linear term favors high-scoring outcomes, while the positive entropy term favors spread. Lowering T weakens the entropy contribution and moves the solution toward an argmax; raising T strengthens it.^[17]

Classification

In a neural network classifier with K mutually exclusive classes, a final linear layer often produces logits

$$\mathbf{z}=W\mathbf{h}+\mathbf{b}.$$

Softmax converts them to a normalized class distribution. The log-odds between classes i and j are

$$\log\frac{p_i}{p_j} = z_i-z_j.$$

When each logit is linear in features, this is the multinomial logistic model. In a deeper network, the same relation holds for the final logits, while the representation h is learned nonlinearly.

The fact that the components are positive and sum to one is necessary for a categorical distribution, but it is not a calibration guarantee. A classifier is calibrated when, over an appropriate population, predictions made with confidence near a value such as 0.8 are correct about 80 percent of the time. Guo and colleagues found substantial miscalibration in the modern neural networks they evaluated and studied post-hoc temperature scaling as a one-parameter correction on held-out data.^[8] A softmax value should therefore be described as a model probability or normalized score unless calibration has been tested for the intended data distribution.

Two classes and the sigmoid

For two logits z_1 and z_2,

$$p_1 = \frac{e^{z_1}}{e^{z_1}+e^{z_2}} = \frac{1}{1+e^{-(z_1-z_2)}}.$$

Thus two-class softmax is exactly the sigmoid function applied to the logit difference, with p_2 = 1 - p_1. Because of shift invariance, one logit can be fixed to zero without changing the represented distribution.

For binary classification, software may use either one sigmoid logit or two softmax logits. The parameterizations are equivalent when their score difference and target conventions match.^[6] They should not be mixed by applying a sigmoid independently to both softmax logits.

Mutually exclusive and multi-label targets

Softmax couples all components through a single normalizer. It is appropriate when one categorical outcome is selected from the listed alternatives, or when a target is a probability distribution over those alternatives. It is generally not the right output for a multi-label task in which several labels may be independently present. Such a task commonly uses one sigmoid and one binary loss per label instead.

This is a modeling distinction, not merely a software preference. Raising the logit of one softmax class necessarily lowers the normalized probabilities of the others. Independent sigmoid outputs do not impose that competition.^[5][6]

Cross-entropy and gradients

For a target distribution y and predicted probabilities p = softmax(z), categorical cross-entropy is

$$L(\mathbf{z},\mathbf{y}) = -\sum_{i=1}^{K}y_i\log p_i.$$

Using the softmax definition,

$$L = -\sum_i y_i z_i + \left(\sum_i y_i\right) \log\left(\sum_j e^{z_j}\right).$$

If y is a valid probability distribution and therefore sums to one,

$$L = -\mathbf{y}^{\mathsf T}\mathbf{z} + \operatorname{LSE}(\mathbf{z}).$$

Differentiating with respect to the logits gives the familiar result

$$\nabla_{\mathbf{z}}L = \mathbf{p}-\mathbf{y}.$$

This compact gradient applies to the ordinary unweighted categorical cross-entropy with a target vector that sums to one. Class weights, label masks, reduction choices, target normalization, or other loss modifications change the exact derivative.

Convexity in logits

For a fixed target distribution, ordinary softmax cross-entropy is convex as a function of the logits because it is log-sum-exp minus a linear term. Its Hessian is

$$\nabla_{\mathbf{z}}^2 L = \operatorname{diag}(\mathbf{p})-\mathbf{p}\mathbf{p}^{\mathsf T},$$

which is positive semidefinite. It is not strictly convex in all logit directions because adding a shared constant leaves both the probabilities and loss unchanged. This statement is about the loss as a function of one logit vector. After the logits are generated by a multilayer network, the training objective as a function of all network parameters is generally not convex.

For a one-hot target y_c = 1, the gradient component for the correct class is p_c - 1, and every other component is p_j. The components sum to zero. If the score gap for the correct class grows without bound, the loss approaches zero and the logit gradient approaches zero. For finite unconstrained logits, however, p_c remains below one, so the hard-label loss does not attain zero. Regularization, finite data, finite precision, early stopping, and parameter constraints affect what happens in an actual trained model.

For a soft target with all components positive, the unconstrained logit optimum satisfies p = y, up to the additive offset. This includes label-smoothed targets and teacher distributions used in distillation. The simple p - y gradient makes clear that smoothing changes the desired probability vector rather than merely scaling the hard-label gradient.

For a hard class label c, the loss reduces to

$$L=-\log p_c = -z_c+\operatorname{LSE}(\mathbf{z}).$$

Libraries normally accept raw logits and fuse log-softmax with the loss. PyTorch's CrossEntropyLoss, for example, expects unnormalized logits and covers both class-index and probability targets.^[4] TensorFlow similarly warns not to feed already-softmaxed values to softmax_cross_entropy_with_logits because the operation applies the normalization internally.^[5]

The fused form is both clearer and more numerically reliable than computing softmax, taking a logarithm, and then applying negative log-likelihood as separate floating-point steps. Backpropagation then propagates the logit gradient through earlier layers.

Numerical computation

The mathematical definition is simple, but a naive implementation

exp(z_i) / sum(exp(z_j))

can fail in finite precision. Large positive logits can overflow during exponentiation, and large negative logits can underflow to zero. Dividing two overflowed quantities can produce NaN even though the exact real-number softmax is well defined.^[3]

Maximum subtraction

Let

$$m=\max_j z_j.$$

Shift invariance gives

$$p_i = \frac{\exp(z_i-m)} {\sum_j\exp(z_j-m)}.$$

Every shifted exponent is at most one, and at least one is exactly one. For finite logits, the denominator is therefore at least one and cannot overflow through exponentiation. Blanchard, Higham, and Higham analyze this shifted algorithm and find it avoids overflow while retaining good accuracy.^[3]

Underflow may still occur for entries far below the maximum. In that case, a tiny exact probability can round to zero. This differs from the exact real-number property that finite-input softmax components are strictly positive. Whether such underflow matters depends on the downstream computation; for log probabilities, computing log-softmax directly is safer than taking the logarithm of a rounded softmax result.

Stable log-softmax

The stable log probability is

$$\log p_i = z_i - \operatorname{LSE}(\mathbf{z}),$$

with

$$\operatorname{LSE}(\mathbf{z}) = m + \log\left(\sum_j e^{z_j-m}\right).$$

This avoids materializing tiny probabilities before taking their logarithms. SciPy documents log_softmax as more accurate than log(softmax(x)) when inputs cause softmax to saturate.^[7] The same principle underlies fused cross-entropy operations.

Masks and infinities

Attention and variable-length batches often exclude positions by replacing their logits with negative infinity before normalization. In exact extended-real arithmetic, exp(-infinity) = 0, so the masked probability is zero. At least one finite, unmasked value must remain in every normalized slice. If an entire slice is masked, the formula has a zero denominator and the desired output is application-specific.

Positive infinity is different. If one score is positive infinity and all others are finite, a limiting argument would place all mass on that component, but direct floating-point maximum subtraction evaluates infinity - infinity, which is NaN. If several scores are positive infinity, an additional tie rule is needed. Software behavior should be checked rather than inferred from the finite-input formula; JAX, for example, documents that a positive-infinity input produces all NaNs in its softmax.^[21]

Axis and shape

Tensor libraries apply softmax along a specified axis. Each slice along that axis gets its own normalizer. Choosing the wrong axis can still produce values that look plausible while normalizing the wrong groups. PyTorch's module exposes a dimension argument and states that every slice along the selected dimension sums to one.^[4] In a batch of class logits shaped [batch, classes], the class axis is normally selected. Attention tensors can require a different final or named axis.

Online normalization

The usual safe algorithm first scans a vector for its maximum, scans again to compute the shifted exponential sum, and then produces normalized outputs. Milakov and Gimelshein derived a recurrence that combines maximum and normalizer updates in one online pass:

$$m_j=\max(m_{j-1},x_j),$$ $$d_j = d_{j-1}e^{m_{j-1}-m_j} + e^{x_j-m_j}.$$

After processing the vector, m_j is the running maximum and d_j is the corresponding shifted normalizer. This reorganization reduces memory traffic and supports fused kernels, while representing the same real-number normalization.^[11] Parallel implementations use blockwise versions of the same rescaling identity.

Reduction order, precision, and kernel fusion can still change the last bits of a floating-point result. "Exact" in algorithm papers commonly means algebraically equivalent to standard attention or softmax, not bit-for-bit identical across all hardware and implementations.

Attention

The scaled dot-product form introduced with the Transformer is

$$\operatorname{Attention}(Q,K,V) = \operatorname{softmax} \left( \frac{QK^{\mathsf T}}{\sqrt{d_k}} \right)V.$$

For each query and attention head, softmax is applied across the eligible key positions. The resulting row is a distribution of nonnegative weights that sums to one, and its weighted sum selects a combination of value vectors. Dividing by sqrt(d_k) controls the scale of dot products before softmax; Vaswani and colleagues argued that unscaled dot products can become large in magnitude and drive softmax into regions with very small gradients.^[10]

Causal attention adds a mask that excludes future key positions, typically by adding negative infinity before softmax. Padding and structural masks use the same principle. The normalization domain is the unmasked keys for one query and one head, not the entire attention tensor.

Standard attention often materializes a score matrix and a probability matrix whose sequence dimensions are quadratic. Flash Attention reorganizes exact scaled dot-product attention into tiles, maintains blockwise softmax statistics, and recomputes selected intermediates during the backward pass. Its primary contribution is reducing transfers between high-bandwidth memory and on-chip storage, not replacing softmax with an approximation.^[12] Performance gains depend on sequence length, head dimension, data type, hardware, and surrounding operations; benchmark results from one configuration are not universal.

Temperature in model workflows

The same mathematical parameter appears in several workflows, but its role is not identical in each.

Sampling

In categorical sampling, dividing logits by T before softmax changes the sampling distribution. Lower temperatures concentrate samples on high-logit outcomes; higher temperatures spread probability more broadly. Greedy argmax is not the same operation as sampling at a small positive temperature. In the unique-maximum case it is the zero-temperature limit, while any actual positive temperature retains nonzero exact probability for every finite logit.

Positive temperature scaling preserves logit order, so it does not change a fixed top-k set when both operations act on logits.^[2] Filters defined from normalized probabilities can behave differently because temperature changes the cumulative probability mass. A system description should therefore state the filter and operation order.

Knowledge distillation

In knowledge distillation, Hinton, Vinyals, and Dean used a higher temperature to reveal relative probabilities among classes that would otherwise receive very small mass. A student model is trained against these softened targets, often together with hard labels.^[9] Their derivation notes that the gradients from the soft-target cross-entropy scale approximately as 1/T^2 at high temperature, motivating multiplication of that loss term by T^2 when combining it with other objectives. This compensation belongs to that distillation formulation; it is not a general rule for every use of temperature.

Post-hoc calibration

Temperature scaling for calibration fits a positive scalar on held-out logits and then applies

$$\mathbf{p} = \operatorname{softmax}(\mathbf{z}/T).$$

Because T is shared across classes and positive, it preserves the predicted class while changing confidence. It must be fitted on data separate from the model's training examples and evaluated on data representative of deployment. The 2017 study by Guo and colleagues found it effective across many of their evaluated classification settings, but that empirical result does not make it a guarantee under distribution shift or for every model.^[8]

Large output spaces

For K already-computed logits, normalization requires evaluating and reducing K components, so ordinary softmax is linear in K. In many language model output layers, however, producing the logits with a matrix multiplication also costs roughly O(dK) per hidden vector, where d is the hidden dimension. It is important to distinguish the cost of the output projection from the additional O(K) normalization.^[14]

Hierarchical softmax

Hierarchical methods replace one flat categorical decision with a path through a tree of binary decisions. Morin and Bengio used a hierarchical probabilistic neural language model in which a word's probability is the product of probabilities along its path.^[13] In a balanced binary tree, a path has O(log K) decisions rather than a flat K-way computation. An unbalanced tree does not provide the same worst-case depth, and the hierarchy changes parameterization and optimization.

Adaptive softmax

Adaptive softmax uses frequency-based clusters and allocates different computational capacity to frequent and rare outcomes. Grave and colleagues designed it as an approximation for large-vocabulary training on GPUs, exploiting unequal word frequencies and matrix-operation efficiency.^[14] Its expected cost depends on the frequency distribution and cluster design. It should not be described as an exact drop-in acceleration of every full-softmax probability calculation.

Online and tiled implementations

Online normalization reduces passes over memory but still processes all included logits. Tiled attention algorithms avoid storing a complete attention matrix by repeatedly rescaling partial maxima and sums. These techniques improve data movement and memory use without changing the asymptotic number of included softmax terms. Claims about speed therefore need to distinguish arithmetic count, memory traffic, kernel launches, and end-to-end runtime.

Reinforcement learning and action selection

In reinforcement learning, softmax can map learned action preferences H_t(a) to a stochastic policy:

$$\pi_t(a) = \frac{e^{H_t(a)}}{\sum_b e^{H_t(b)}}.$$

Sutton and Barto present this rule in their section on gradient bandit algorithms. Only relative preferences matter, and adding the same constant to every action preference leaves the policy unchanged.^[15] A reward-baseline update can increase the selected action's preference when reward exceeds the baseline and decrease it when reward falls below the baseline.

This use does not make softmax an exploration guarantee. A sharply concentrated policy may assign extremely small probability to alternatives, while a high temperature may explore broadly at the cost of selecting lower-preference actions more often. Policy behavior also depends on how preferences are learned, whether invalid actions are masked, and whether temperature changes over time.

The derivative identity used in policy gradients follows directly from softmax:

$$\frac{\partial\log\pi(a)}{\partial z_j} = \delta_{aj}-\pi(j).$$

It gives a selected-action term minus the policy expectation. The broader correctness and variance of a policy-gradient estimator require additional assumptions beyond this local derivative.

Alternatives and extensions

Softmax is dense for finite inputs: every component is strictly positive in exact arithmetic. Several alternatives change that behavior or use softmax inside a larger construction.

Sparsemax and entmax

Sparsemax projects a score vector onto the probability simplex in Euclidean distance. It can assign exact zero probability to low-scoring components, unlike finite-input softmax.^[16] The mapping is piecewise linear and differentiable almost everywhere, but its loss and derivatives differ from ordinary softmax cross-entropy.

The alpha-entmax family generalizes entropy-regularized prediction. It includes softmax at alpha = 1 and sparsemax at alpha = 2; values greater than one can produce sparse distributions.^[17] Sparsity can be useful when exact zeros are part of the intended inductive bias, but it is not automatically more accurate or more efficient. Computing the mapping, choosing the associated loss, and handling support changes all matter.

Gumbel-Softmax and the Concrete distribution

Gumbel-Softmax perturbs log class probabilities with independent Gumbel noise and applies a temperature-controlled softmax:

$$y_i = \frac{\exp((\log\pi_i+g_i)/T)} {\sum_j\exp((\log\pi_j+g_j)/T)}.$$

For positive temperature, this produces a differentiable random vector in the simplex. As temperature approaches zero, samples approach one-hot categorical samples. Jang, Gu, and Poole introduced this estimator as a differentiable relaxation for categorical latent variables.^[18] Maddison, Mnih, and Teh independently developed the corresponding Concrete distribution and its density.^[19]

At nonzero temperature, the relaxed sample is not identical to a categorical sample. Lower temperature makes it closer to one-hot but can make gradients less well behaved; higher temperature makes it smoother but increases the discrepancy from the discrete variable. Straight-through variants use a hard value in the forward pass and a relaxed derivative in the backward pass, which introduces a biased gradient estimator.

Mixture of softmaxes and the softmax bottleneck

In a standard neural language-model output layer, context vectors and output embeddings form a low-rank logit matrix before row-wise softmax. Yang and colleagues showed that, after accounting for softmax's row-shift invariance, this factorization can limit the rank of the representable log-probability matrix. They called this the softmax bottleneck and proposed a mixture of softmax distributions to increase expressiveness.^[20]

The result is about a particular factorized output parameterization across many contexts, not a claim that a single softmax vector cannot represent an arbitrary strictly positive categorical distribution. For one vector, choosing logits proportional to log probabilities represents any interior point of the simplex. The bottleneck emerges from sharing a low-dimensional factorization across contexts.

Software interfaces

Common numerical libraries expose softmax with an explicit normalization axis and usually provide a separate log-softmax or fused loss.

Scroll sideways for more →

Interfaces evolve, so code should specify the library version, selected axis, input dtype, target convention, reduction, and mask behavior when reproducibility matters.

Common misconceptions and limitations

"Softmax outputs confidence"

Softmax outputs normalized model scores. Calling the largest component "confidence" is common, but calibration is an empirical property relative to outcomes, not a consequence of normalization. Distribution shift, model misspecification, overfitting, and adversarial or out-of-domain inputs can all break a confidence interpretation.

"The probabilities are independent"

They are coupled by the normalizer and must sum to one. Increasing one logit with all others fixed raises its probability and lowers the rest. This is desirable for mutually exclusive outcomes and inappropriate when labels should vary independently.

"Softmax is a smooth argmax"

Softmax is smooth and preserves the maximizing indices, so the phrase is useful informally. It is not a numerical approximation to the scalar max in the same codomain: softmax returns a probability vector. Log-sum-exp is the smooth scalar approximation to a maximum, while softmax is its gradient. A softmax-weighted average of values is sometimes called a soft argmax, but that is a separate composition.

"Subtracting the maximum changes the probabilities"

In exact real arithmetic it does not, because of shift invariance. In floating-point arithmetic, the shifted expression is intentionally used because it avoids overflow and usually improves accuracy. Different reduction orders can still produce small rounding differences.

"Exact softmax never returns zero"

For finite real inputs, that statement is true. A floating-point implementation can underflow a tiny exponential to zero, and an explicit negative-infinity mask is intended to create zero. Mathematical and implementation claims must therefore be kept separate.

"One softmax operation is the whole output-layer cost"

Normalization is linear in the number of logits after they exist. In a large-vocabulary model, generating those logits through an output projection can be more expensive than normalization. Hierarchical, adaptive, sampled, and hardware-aware methods address different parts of that cost and make different accuracy or parameterization tradeoffs.

History and terminology

Normalized exponentials appeared in statistical and physical models before modern neural networks. In neural-network literature, John S. Bridle's 1989 NeurIPS paper described a normalized exponential output transformation and wrote that the authors liked to call it "soft max."^[1] That source supports an early documented use of the term; it does not establish that no earlier author used the same function or wording.

Spellings include "softmax," "soft-max," and "soft max." Contemporary software generally uses softmax. "Softargmax" should not be treated as an exact synonym without context: it often means taking an expectation or weighted average after softmax, which has a different output and purpose.

See also

• Attention
• Gradient Descent
• Natural Language Processing

References

1. ^Bridle, J. S. (1989). "Training Stochastic Model Recognition Algorithms as Networks can Lead to Maximum Mutual Information Estimation of Parameters." *Advances in Neural Information Processing Systems 2*. proceedings.neurips.cc/...24f4333c7658a0e-Abstract
2. ^Gao, B., and Pavel, L. (2017). "On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning." arXiv:1704.00805. arxiv.org/...1704.00805
3. ^Blanchard, P., Higham, D. J., and Higham, N. J. (2021). "Accurately Computing the Log-Sum-Exp and Softmax Functions." *IMA Journal of Numerical Analysis*, 41(4), 2311-2330. doi.org/...draa038
4. ^PyTorch. "Softmax" and "CrossEntropyLoss" documentation. Accessed 2026-07-28. docs.pytorch.org/...torch.nn.Softmax and docs.pytorch.org/...torch.nn.CrossEntropyLoss
5. ^TensorFlow. "`tf.nn.softmax_cross_entropy_with_logits`" documentation. Accessed 2026-07-28. tensorflow.org/...softmax_cross_entropy_with_logits
6. ^TensorFlow. "`tf.nn.sigmoid_cross_entropy_with_logits`" documentation. Accessed 2026-07-28. tensorflow.org/...sigmoid_cross_entropy_with_logits
7. ^SciPy. "`scipy.special.log_softmax`" documentation. Accessed 2026-07-28. docs.scipy.org/...scipy.special.log_softmax
8. ^Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). "On Calibration of Modern Neural Networks." *Proceedings of Machine Learning Research*, 70, 1321-1330. proceedings.mlr.press/...guo17a
9. ^Hinton, G., Vinyals, O., and Dean, J. (2015). "Distilling the Knowledge in a Neural Network." arXiv:1503.02531. arxiv.org/...1503.02531
10. ^Vaswani, A., Shazeer, N., Parmar, N., et al. (2017). "Attention Is All You Need." *Advances in Neural Information Processing Systems 30*. proceedings.neurips.cc/...bd053c1c4a845aa-Abstract
11. ^Milakov, M., and Gimelshein, N. (2018). "Online Normalizer Calculation for Softmax." arXiv:1805.02867. arxiv.org/...1805.02867
12. ^Dao, T., Fu, D. Y., Ermon, S., Rudra, A., and Re, C. (2022). "FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness." *Advances in Neural Information Processing Systems 35*. arxiv.org/...2205.14135
13. ^Morin, F., and Bengio, Y. (2005). "Hierarchical Probabilistic Neural Network Language Model." *Proceedings of Machine Learning Research*, R5, 246-252. proceedings.mlr.press/...morin05a
14. ^Grave, E., Joulin, A., Cisse, M., Grangier, D., and Jegou, H. (2017). "Efficient Softmax Approximation for GPUs." *Proceedings of Machine Learning Research*, 70, 1302-1310. proceedings.mlr.press/...grave17a
15. ^Sutton, R. S., and Barto, A. G. (2018). *Reinforcement Learning: An Introduction*, second edition, Section 2.8. MIT Press. incompleteideas.net/...the-book-2nd
16. ^Martins, A. F. T., and Astudillo, R. F. (2016). "From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification." *Proceedings of Machine Learning Research*, 48, 1614-1623. proceedings.mlr.press/...martins16
17. ^Peters, B., Niculae, V., and Martins, A. F. T. (2019). "Sparse Sequence-to-Sequence Models." *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, 1504-1519. aclanthology.org/P19-1146
18. ^Jang, E., Gu, S., and Poole, B. (2017). "Categorical Reparameterization with Gumbel-Softmax." *International Conference on Learning Representations*. openreview.net/forum
19. ^Maddison, C. J., Mnih, A., and Teh, Y. W. (2017). "The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables." *International Conference on Learning Representations*. openreview.net/forum
20. ^Yang, Z., Dai, Z., Salakhutdinov, R., and Cohen, W. W. (2018). "Breaking the Softmax Bottleneck: A High-Rank RNN Language Model." *International Conference on Learning Representations*. openreview.net/forum
21. ^JAX. "`jax.nn.softmax`" documentation. Accessed 2026-07-28. docs.jax.dev/...jax.nn.softmax