Multinomial regression

See also: Machine learning terms

Multinomial regression is a statistical model that predicts which one of K possible categories an observation belongs to, given a vector of input features. It generalises logistic regression, which handles two classes, to the case where K is greater than two: the model assigns each class its own linear score, then turns those scores into a probability distribution using the softmax function, and the class with the highest probability is the prediction. ^[1][2] The same mathematical form is the final layer of essentially every modern classification neural network, from image classifiers to the next-token head of large language models: a linear layer feeding a softmax with a cross-entropy loss is exactly multinomial regression over learned features. ^[14]

The model goes by several names, all of which describe the same algorithm: multinomial logistic regression, softmax regression, polytomous logistic regression, and the maximum entropy classifier (often shortened to MaxEnt) in the natural language processing literature. Discrete-choice economists call the same model the multinomial logit, and a closely related variant is McFadden's conditional logit. All of these names refer to a generalized linear model with a multinomial response distribution and a softmax link function. ^[1]

Multinomial regression sits at the boundary between classical statistics and modern machine learning. Statisticians use it for inference: estimating odds ratios, computing confidence intervals, and testing whether a predictor matters. Machine learning practitioners use it as a strong baseline for multinomial classification and as the final layer of nearly every deep neural network that produces a categorical output, including the next-token head of large language models.

Model definition

Let $x \in \mathbb{R}^d$ be a feature vector and let y be a categorical label taking values in $\{1, 2, \ldots, K\}$. Multinomial regression models the conditional probability of each class as

$$P(y = k \mid x) = \frac{\exp(w_k \cdot x + b_k)}{\sum_{j=1}^{K} \exp(w_j \cdot x + b_j)}$$

for $k = 1, \ldots, K$. Each class k has its own weight vector $w_k \in \mathbb{R}^d$ and its own bias $b_k \in \mathbb{R}$. The denominator (the partition function) ensures the K probabilities sum to one. Equivalently, if $z_k = w_k \cdot x + b_k$ is the linear score (or logit) for class k, then the predicted probability is the softmax of the score vector $z = (z_1, \ldots, z_K)$. ^[1][2]

The decision boundary between any two classes k and l is the set of inputs where $w_k \cdot x + b_k = w_l \cdot x + b_l$, which is a hyperplane. So multinomial regression carves the input space into K convex polytopes, one per class. The boundaries are linear in x, even though the probability surface is smooth.

Identifiability and the reference category

The parameterisation above is overcomplete. Adding the same constant vector c to every weight vector (replacing each $w_k$ with $w_k + c$) leaves the probabilities unchanged, because the c terms cancel inside the softmax. The same applies to the biases. With no further constraint there are infinitely many parameter settings that give identical predictions, which makes maximum-likelihood estimates undefined. ^[1]

Two conventions resolve this:

Scroll sideways for more →

The reference-category form has a clean interpretation: each $w_k$ tells you how feature x shifts the log-odds of class k versus the reference class. The symmetric form is more convenient for software, especially when the model is a layer inside a neural network and weight decay is already in the loss.

How is multinomial regression trained?

Multinomial regression is trained by maximum likelihood estimation. Given a dataset $\{(x_i, y_i)\}$ for $i = 1, \ldots, n$, the log-likelihood is

$$L(\theta) = \sum_{i=1}^{n} \sum_{k=1}^{K} \mathbb{1}\{y_i = k\} \cdot \log P(y_i = k \mid x_i)$$

where $\theta = \{w_k, b_k\}$ collects all weights and biases. Maximising this likelihood is exactly the same as minimising the average categorical cross-entropy loss. The cross-entropy view is dominant in deep learning; the maximum-likelihood view is dominant in statistics. They are the same objective up to a sign and a constant. ^[1][14]

The gradient with respect to w_k is

$$\frac{\partial L}{\partial w_k} = \sum_i \left(\mathbb{1}\{y_i = k\} - P(y_i = k \mid x_i)\right) \cdot x_i$$

which has the elegant form (label minus prediction) times input. This is the same gradient that backpropagates through a softmax-plus-cross-entropy layer in a neural network, and it is one reason this combination is so popular: the gradient is numerically stable and easy to compute. ^[14]

Unlike ordinary least squares, multinomial regression has no closed-form solution. The likelihood is, however, concave in the parameters (strictly concave once a regulariser is added), so any local optimiser converges to the unique global maximum. ^[1] Common algorithms include:

Scroll sideways for more →

Thomas Minka's 2003 comparison of optimisers for logistic regression found that conjugate gradient and quasi-Newton methods (such as L-BFGS) consistently dominated the naive iterative scaling that had been the standard in the maximum-entropy NLP community; on a typical dataset, the cost gap between the fastest optimiser and the slowest was more than two orders of magnitude. ^[9] Modern libraries reflect that finding: scikit-learn's default solver is L-BFGS, ^[11] and even when training enormous neural networks the final softmax layer is fit with the same kind of gradient steps. ^[14]

Regularization

Without a penalty term, multinomial regression on linearly separable data has weights that diverge to infinity, which is one symptom of overfitting. Regularization fixes this by adding a penalty on the size of the weights. The three standard choices map directly to Bayesian priors on the weights: ^[1]

Scroll sideways for more →

L2 is the workhorse and the default in scikit-learn, statsmodels, and almost every deep-learning framework (where it is usually called weight decay). ^[14] L1 is useful when most features are irrelevant and you want feature selection as a by-product of fitting. Elastic net, introduced by Zou and Hastie in 2005, is preferred when correlated features make pure L1 unstable. ^[10]

Connections and equivalent models

Multinomial regression keeps surfacing under different names in different fields, because the same mathematical object is genuinely useful in many places.

• Generalized linear model. The multinomial response distribution combined with the softmax (multinomial logit) link function gives a GLM. This is the framework that connects logistic regression, Poisson regression, and ordinary least squares as instances of the same template. ^[1]
• Log-linear model. The softmax can be rewritten as a log-linear model over (x, y) pairs. This is how the model is presented in many statistics textbooks on contingency tables.
• Maximum entropy classifier. Adam Berger and colleagues at IBM showed in 1996 that fitting a log-linear model under feature-expectation constraints is equivalent to choosing the maximum-entropy distribution consistent with those constraints. The dual of the maximum-entropy problem is the maximum-likelihood problem for multinomial regression. ^[6] NLP researchers used the MaxEnt name for the next decade or so. Adwait Ratnaparkhi's 1996 part-of-speech tagger built on the same framework and was the dominant POS approach until neural networks took over. ^[7]
• Conditional random field. A linear-chain CRF is structurally a multinomial regression at every position, with extra terms that couple adjacent labels. CRFs were the standard sequence-labelling tool from 2001 (Lafferty, McCallum, Pereira) until BiLSTM-CRF and then transformer architectures replaced them. ^[8]
• Conditional logit. Daniel McFadden's 1974 conditional logit model is multinomial regression in which the features depend on both the chooser and the alternative being chosen. ^[4] McFadden shared the 2000 Nobel Memorial Prize in Economic Sciences with James Heckman, cited "for his development of theory and methods for analyzing discrete choice," work that underpinned modern transportation demand forecasting and consumer choice analysis. ^[12]
• Final layer of a neural network. A softmax over logits with a cross-entropy loss is exactly multinomial regression on whatever features the rest of the network has produced. Image classifiers, speech recognisers, and large language models all end this way. ^[14] In a transformer language model the last layer is multinomial regression over a vocabulary of tens or hundreds of thousands of tokens.

How does multinomial regression differ from one-vs-rest?

A common alternative to fitting a single multinomial model is to fit K independent binary logistic regressions, each one separating its class from all the rest. Both approaches give a per-class score, but they have different statistical properties.

Scroll sideways for more →

For most applications with moderate K, the joint multinomial fit produces better calibrated probabilities and slightly higher accuracy. One-vs-rest is more attractive when K is enormous (millions of categories) and you need to parallelise across many machines, or when you want to plug a binary-only library into a multiclass problem.

Statistical inference

When multinomial regression is used as a statistical model rather than a black-box predictor, several inference tools are standard:

• Standard errors and Wald tests. Asymptotic standard errors come from the inverse Fisher information (the inverse Hessian of the negative log-likelihood at the optimum). Dividing each coefficient by its standard error gives a Wald statistic, which is approximately normal under the null hypothesis that the coefficient is zero.
• Likelihood ratio tests. To test whether a group of coefficients can be set to zero, fit the model with and without those coefficients and compare twice the difference in log-likelihoods to a chi-squared distribution with degrees of freedom equal to the number of constrained parameters.
• Odds ratios. For each coefficient $w_{k,m}$ (feature m, class k versus the reference class K), $\exp(w_{k,m})$ is the multiplicative change in the odds of class k versus class K when feature m increases by one unit, holding other features fixed. Confidence intervals on odds ratios are usually obtained by exponentiating Wald intervals on the coefficients themselves.
• McFadden's pseudo R-squared. Defined as 1 minus the ratio of the fitted model's log-likelihood to the null model's log-likelihood. McFadden himself wrote that "values of .2 to .4" for the statistic "represent excellent fit." ^[13] The value is not directly comparable to OLS R-squared and tends to be much smaller than what people expect from a linear regression context. ^[3]

Implementations

Multinomial regression is in essentially every statistics and machine-learning library. The most widely used:

Scroll sideways for more →

Code example: scikit-learn

from sklearn.datasets import load_iris
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

clf = LogisticRegression(solver="lbfgs", C=1.0, max_iter=1000)
clf.fit(X_train, y_train)

print("accuracy:", clf.score(X_test, y_test))
print("per-class probabilities for first test point:", clf.predict_proba(X_test[:1]))

With three iris species this is a textbook three-class softmax fit. predict_proba returns the softmax probabilities directly, and clf.coef_ has shape $(3, 4)$: one weight vector per class, one entry per input feature.

How is multinomial logit used in discrete choice modelling?

In econometrics, multinomial logit is the workhorse for modelling choice among finite alternatives, including which mode of transport someone takes to work, which brand of cereal they buy, or which job offer they accept. McFadden's 1974 conditional logit allowed the features to depend on the alternatives themselves (price of each option, travel time of each route), which was a major step beyond the basic multinomial logit where only the chooser's attributes mattered. ^[4]

The basic multinomial logit imposes the independence of irrelevant alternatives (IIA): the ratio of probabilities between any two alternatives does not depend on what other alternatives are available. McFadden's famous red-bus blue-bus example shows why this is restrictive. If a commuter is split 50/50 between car and bus, and you add an identical bus painted a different colour, IIA forces the model to predict 1/3, 1/3, 1/3 instead of the more realistic 1/2, 1/4, 1/4. ^[5] Nested logit and mixed logit relax IIA by grouping similar alternatives or letting coefficients vary across the population.

What are the limitations of multinomial regression?

Multinomial regression has several real limitations that are worth being honest about.

The decision boundary between any two classes is linear in the input features. If the true boundary is curved (a parabola, a circle, anything nonlinear) the model cannot fit it without explicit feature engineering, kernel methods, or a nonlinear model on top.

The IIA assumption is baked into the basic model. When alternatives are close substitutes, IIA produces predictions that are noticeably wrong. ^[4] The practical fix is either to use a richer model (nested logit, mixed logit, neural network) or to be careful about how the choice set is defined.

Like all maximum-likelihood methods, multinomial regression can overfit when d is large relative to n, especially when classes are nearly separable. Without regularisation the optimiser will happily push weights to infinity to drive the softmax probabilities to one and zero. L2 regularisation prevents this but introduces a hyperparameter that must be tuned by cross-validation.

Finally, multinomial regression is symmetric in the way it treats classes, so it does not exploit ordinal structure. If your labels have a natural order (1 to 5 stars, low/medium/high), an ordinal model usually fits better and uses fewer parameters.

How is multinomial regression used in modern deep learning?

For classification problems where the input is raw text, image pixels, or audio waveforms, multinomial regression on hand-crafted features has been displaced by deep neural networks that learn their own representations. But the displacement is partial. Almost every classifier in production today, including the largest neural networks, ends with multinomial regression on top of learned features.

The final layer of a classifier built with TensorFlow or PyTorch is a Dense(K) (or Linear(d, K)) followed by a softmax-cross-entropy loss. That is multinomial regression. ^[14] The features it operates on are the activations of the previous layer, learned end-to-end with the rest of the network. The same arithmetic that Berger and colleagues used for maximum-entropy text classification in 1996 is what trains the final layer of GPT-style language models in 2026. ^[6]

Language modelling is the most striking example. A transformer language model treats next-token prediction as a multinomial regression problem over the entire vocabulary, which typically runs to tens or hundreds of thousands of tokens: for example, GPT-4's cl100k_base tokenizer defines a vocabulary of 100,277 tokens, ^[15] and Llama 3 uses a 128,256-token vocabulary. ^[16] The model produces a logit vector of size $V$ (vocabulary size), the softmax converts it into a probability distribution, and training minimises the cross-entropy between that distribution and the one-hot vector for the actual next token. Sampling from a language model is sampling from a multinomial regression's predicted probabilities. Greedy decoding is taking the argmax. Temperature scaling is dividing the logits by a constant before the softmax. All of these tricks are operations on a multinomial regression that happens to have a giant feature extractor in front of it.

Multinomial regression is also still useful in its own right. It is fast to train, easy to interpret, naturally calibrated, and gives a strong baseline that more complex models have to beat. For tabular data with modest dimensionality it often loses very little accuracy to gradient boosting, and it remains the default first model for many practical classification problems.

Explain like I'm 5 (ELI5)

Imagine you have a bag of candies in different colours, and you want to guess each candy's colour from its shape and size. For two colours, you would draw one line that separates them. For ten colours, you draw ten lines, each one giving a score for one colour. To turn the scores into probabilities (so they all add up to one and the highest score wins), you use a function called softmax. That whole recipe is multinomial regression. You teach the lines by showing many candies whose colours you know, and the computer adjusts the lines until the probabilities for the right colours are as high as possible.

References

1. Wikipedia. "Multinomial logistic regression." https://en.wikipedia.org/wiki/Multinomial_logistic_regression ↩
2. Wikipedia. "Softmax function." https://en.wikipedia.org/wiki/Softmax_function ↩
3. Wikipedia. "Pseudo-R-squared." https://en.wikipedia.org/wiki/Pseudo-R-squared ↩
4. McFadden, Daniel. "Conditional logit analysis of qualitative choice behavior." In P. Zarembka (ed.), Frontiers in Econometrics, Academic Press, 1974, pp. 105 to 142. https://eml.berkeley.edu/reprints/mcfadden/zarembka.pdf ↩
5. McFadden, Daniel. Nobel lecture: "Economic Choices." 2000. https://eml.berkeley.edu/~mcfadden/nobel/final-nobel.pdf ↩
6. Berger, Adam L., Stephen A. Della Pietra, and Vincent J. Della Pietra. "A maximum entropy approach to natural language processing." Computational Linguistics 22(1), 1996. https://aclanthology.org/J96-1002.pdf ↩
7. Ratnaparkhi, Adwait. "A maximum entropy model for part-of-speech tagging." EMNLP, 1996. https://aclanthology.org/W96-0213.pdf ↩
8. Lafferty, John, Andrew McCallum, and Fernando Pereira. "Conditional random fields: Probabilistic models for segmenting and labeling sequence data." ICML, 2001. ↩
9. Minka, Thomas P. "A comparison of numerical optimizers for logistic regression." 2003. https://tminka.github.io/papers/logreg/minka-logreg.pdf ↩
10. Zou, Hui, and Trevor Hastie. "Regularization and variable selection via the elastic net." Journal of the Royal Statistical Society B 67(2), 2005, pp. 301 to 320. ↩
11. scikit-learn developers. "sklearn.linear_model.LogisticRegression." https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html ↩
12. The Royal Swedish Academy of Sciences. "The Sveriges Riksbank Prize in Economic Sciences in Memory of Alfred Nobel 2000." https://www.nobelprize.org/prizes/economic-sciences/2000/popular-information/ ↩
13. McFadden, Daniel. "Quantitative Methods for Analyzing Travel Behaviour of Individuals: Some Recent Developments." Cowles Foundation Discussion Paper No. 474, Yale University, 1977. https://elischolar.library.yale.edu/cowles-discussion-paper-series/707/ ↩
14. Goodfellow, Ian, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. https://www.deeplearningbook.org/ ↩
15. OpenAI. "tiktoken." GitHub. https://github.com/openai/tiktoken ↩
16. Meta. "Llama 3 Model Card." GitHub, 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md ↩