Multinomial classification

See also: Machine learning terms

What is multinomial classification?

Multinomial classification, also called multiclass or multi-class classification, is the supervised learning task of assigning each input to exactly one of K possible classes, where K is greater than two. It generalises binary classification (which has only two outcomes) to three or more mutually exclusive categories. The scikit-learn documentation defines it directly: "Multiclass classification is a classification task with more than two classes. Each sample can only be labeled as one class." ^[1] Examples include classifying a handwritten digit as one of 0 through 9, predicting which species of iris a flower belongs to, or labelling an ImageNet photograph with one of 1,000 object categories.

Formally, the classifier learns a function $f(x) = y$ that maps a feature vector $x$ to a single label $y$ in $\{1, 2, \ldots, K\}$. The task sits at the heart of machine learning: many real-world problems have more than two outcomes, so binary methods alone are not enough. Standard approaches either build a single model that outputs K probabilities directly, or wrap a binary classification algorithm with a meta-strategy such as one-vs-rest or one-vs-one. The choice depends on the algorithm, the number of classes, the size of the dataset, and the computational budget. ^[1]

How is the problem formulated?

Given a training set $\{(x_i, y_i)\}$ for $i = 1$ to $n$, where $x_i$ is a feature vector in $\mathbb{R}^d$ and $y_i$ is an integer label in $\{1, 2, \ldots, K\}$, the goal is to learn a hypothesis $h$ that predicts a label for any new input $x$. Most classifiers learn a scoring function $s_k(x)$ for each class $k$, then predict the argmax:

$$\hat{y} = \arg\max_k s_k(x)$$

When the scores can be interpreted as probabilities, the model also returns a posterior $P(y = k \mid x)$. Probabilistic outputs matter for downstream tasks such as ranking, calibration, abstention, and decision theory under asymmetric costs.

What is the difference between multiclass and multi-label classification?

Multinomial classification is often confused with adjacent problems. The differences matter because the choice of loss, evaluation metric, and output layer all change. The key distinction is exclusivity: in multiclass classification a sample gets exactly one label, whereas in multi-label classification it can carry several at once. The scikit-learn documentation frames multi-label as "a classification task labeling each sample with m labels from n_classes possible classes, where m can be 0 to n_classes inclusive" and describes it as "predicting properties of a sample that are not mutually exclusive." ^[1] Tagging a news article as both "politics" and "finance" is multi-label; sorting a single digit image into one of ten buckets is multiclass.

Scroll sideways for more →

How do you turn binary classifiers into multiclass?

A classifier can either treat K classes natively or reduce the problem to a series of binary subproblems. The four main strategies are summarised below.

Scroll sideways for more →

The one-vs-rest approach (also called one-vs-all) trains K classifiers, where the k-th classifier learns to separate class k from the union of all other classes. Prediction picks the class whose classifier produces the highest score. The scikit-learn documentation calls it "the most commonly used strategy and a fair default choice," noting that "only n_classes classifiers are needed" and that each class "is represented by one and only one classifier," which makes the model easy to interpret. ^[1] It is the default meta-strategy in many libraries because it scales linearly with K and produces a per-class confidence score.

One-vs-one trains $K(K - 1) / 2$ classifiers, each on the subset of training data belonging to two specific classes. At test time, every classifier votes for one of its two classes, and the class with the most votes wins. Because it requires $O(K^2)$ submodels, it is usually slower than one-vs-rest, but scikit-learn notes that "this method may be advantageous for algorithms such as kernel algorithms which don't scale well with n_samples," since each individual problem uses far less data. ^[1] This makes it a common default for kernel methods such as support vector machines, where the cost of training kernel matrices grows superlinearly with sample size.

Error-correcting output codes were introduced by Thomas Dietterich and Ghulum Bakiri in 1995. ^[2] Each of the K classes is assigned a binary codeword of length L, and L binary classifiers are trained to predict each bit. At test time, the predicted bits form a codeword that is decoded to the nearest class codeword by Hamming distance. The redundancy in the code lets the system recover when individual binary classifiers are wrong, which often improves generalisation. ^[2]

Which algorithms handle multiclass natively?

Most modern algorithms support multinomial classification directly without reduction to binary subproblems. The scikit-learn library lists decision trees, naive Bayes (Gaussian and Bernoulli), k-nearest neighbours, random forests, logistic regression, linear and quadratic discriminant analysis, and multilayer perceptron neural networks among the estimators that are "inherently multiclass" and need no meta-estimator. ^[1]

Scroll sideways for more →

Koby Crammer and Yoram Singer published their multiclass kernel-based vector machine formulation in the Journal of Machine Learning Research in 2001 (volume 2, pages 265-292). ^[3] Unlike earlier multiclass support vector machines, which decomposed the problem into independent binary tasks, their algorithm casts the entire K-class problem as a single constrained optimisation problem with a quadratic objective. Solving it through the dual yields a fixed-point iteration that handles many classes efficiently. ^[3]

What is softmax regression?

Softmax regression, also called multinomial regression or multinomial logistic regression, is the canonical native multiclass model. For each class $k$ it learns a weight vector $w_k$ and bias $b_k$, then defines:

$$P(y = k \mid x) = \frac{\exp(w_k \cdot x + b_k)}{\sum_j \exp(w_j \cdot x + b_j)}$$

The denominator normalises the scores into a valid probability distribution, so the K outputs are non-negative and sum to 1. ^[10] The model is trained by maximising the log-likelihood of the observed labels, which is equivalent to minimising the categorical cross-entropy loss:

$$L = -\sum_i \sum_k \mathbf{1}\{y_i = k\} \log P(y_i = k \mid x_i)$$

When $K = 2$, softmax regression reduces to ordinary logistic regression. The same softmax + cross-entropy combination is the default output layer for almost every modern deep classifier, from MNIST digit recognisers to large language models. ^[10]

Which loss functions are used for multiclass problems?

The choice of loss controls what the model optimises and how it handles imbalance, noise, and overconfidence.

Scroll sideways for more →

Label smoothing was introduced by Christian Szegedy and colleagues in the 2016 Inception-v3 paper "Rethinking the Inception Architecture for Computer Vision." ^[4] The technique replaces the one-hot target with a smoothed distribution, which discourages the network from producing extremely confident logits. It tends to improve generalisation and calibration on large vision benchmarks.

Focal loss was introduced by Tsung-Yi Lin and colleagues in 2017 in "Focal Loss for Dense Object Detection." ^[5] Although the original target was object detection with a long-tail of background examples, the modulating factor $(1 - p_t)^\gamma$ works for any imbalanced multiclass problem by reducing the loss contribution of well-classified examples and concentrating gradient on hard ones.

How is multiclass performance evaluated?

No single number captures multiclass performance, especially when classes are imbalanced. Practitioners typically report several complementary metrics.

Scroll sideways for more →

Macro-averaging treats every class as equally important, which matters when the rare classes are the ones you care about. Micro-averaging weights by instance count, so the dominant class drives the score. Weighted averaging is a compromise that uses per-class support as weights. The convention is documented in the scikit-learn classification report and is widely reused in research papers. ^[9]

How is class imbalance handled?

Multiclass datasets often have skewed label distributions. ImageNet-21K's tail contains classes with only a few hundred images, and many real text classification problems have one dominant category and a long list of rare ones. ^[6] Common remedies include:

• Stratified train, validation, and test splits so every class is represented in proportion.
• Class weights in the loss, where each class's contribution is scaled by the inverse of its frequency.
• Resampling, by oversampling rare classes or undersampling common ones.
• Synthetic minority oversampling (SMOTE) for tabular data.
• Focal loss for extreme tail behaviour. ^[5]
• Hierarchical softmax or sampled softmax when K is so large that computing the full normalisation is impractical.

For more on these techniques and their trade-offs, see class imbalance.

What are the famous multiclass benchmarks?

Progress in multinomial classification has been driven by a handful of canonical datasets.

Scroll sideways for more →

The ImageNet Large Scale Visual Recognition Challenge popularised top-1 and top-5 accuracy as the headline scores. ^[11] Top-5 accuracy counts a prediction as correct if the true label appears in the model's five highest-scoring guesses, which makes sense for a 1,000-class problem where neighbouring categories (such as different dog breeds) are visually almost identical. AlexNet hit a top-5 error of 15.3 percent in 2012, ResNet drove it to 3.57 percent in 2015, and Squeeze-and-Excitation Networks reached 2.25 percent in 2017. ^[11] The challenge ended after 2017, with organisers noting that the headline benchmark had been effectively solved.

How is multiclass classification implemented in practice?

Most machine learning libraries handle K-class problems with little extra code.

Scroll sideways for more →

A minimal scikit-learn example using the iris dataset, which has three classes, looks like this:

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report, confusion_matrix

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, stratify=y, random_state=0
)

clf = LogisticRegression(multi_class='multinomial', max_iter=1000)
clf.fit(X_train, y_train)

y_pred = clf.predict(X_test)
print(confusion_matrix(y_test, y_pred))
print(classification_report(y_test, y_pred, digits=3))

The classification_report prints per-class precision, recall, F1, and support, plus macro and weighted averages, which together give a much fuller picture than accuracy alone. ^[9]

How does multiclass classification show up in modern deep learning?

Multinomial classification underpins much of modern deep learning, often in places that look at first like very different tasks.

Next-token prediction in large language models is a multinomial classification problem over the model's vocabulary, which typically has tens of thousands of tokens (50,257 for GPT-2, 100,256 for OpenAI's o200k base, 128,000-plus for some Llama variants). At every position, the model produces a vector of logits, applies softmax, and is trained with cross-entropy against the next true token. The same loss that powers iris classification scales up to train trillion-parameter models on web-scale text. ^[10]

Zero-shot image classification with CLIP is another reformulation. CLIP encodes the image and a list of candidate class names (often phrased as "a photo of a {label}") into a shared embedding space, then picks the class with the highest cosine similarity. ^[7] This effectively turns any vocabulary into a multiclass classifier without retraining, which is one reason CLIP-style models opened up flexible recognition systems.

Retrieval-augmented systems and recommender systems often dress up as ranking or retrieval problems, but at the prediction layer they are usually multiclass softmaxes over a discrete catalogue of items.

What are the limitations of multiclass classification?

The softmax + cross-entropy recipe is robust, but it has known weaknesses.

• The softmax denominator scales linearly with K, which becomes a bottleneck for extreme classification problems with millions of classes. Sampled softmax, hierarchical softmax, and noise-contrastive estimation are common workarounds.
• Vanilla classifiers ignore relationships between classes. "Husky" and "malamute" are penalised the same as "husky" and "airliner," which is rarely what you want. Hierarchical or label-embedding methods address this when class relationships are known.
• Confident wrong predictions are common with cross-entropy training. Label smoothing, temperature scaling, and post-hoc calibration help. ^[4]
• Long-tailed distributions hurt rare classes. Resampling, class-balanced losses, and decoupled training schedules each address parts of the problem. ^[5]
• Open-set recognition is not handled by softmax, which assigns probability mass to one of the K seen classes even for genuinely unknown inputs. Specialised methods such as OpenMax, energy-based detectors, and threshold tuning are used in practice.

Explain like I'm 5

Imagine a basket of fruits with apples, bananas, and oranges. Sorting each piece into the right pile is multinomial classification. The computer looks at lots of labelled examples until it learns what each fruit looks like, then it can pick the right pile for a new piece it has never seen before. The only difference from a yes-or-no question is that there are more than two piles to choose from.

References

1. scikit-learn developers. "Multiclass and multioutput algorithms." https://scikit-learn.org/stable/modules/multiclass.html ↩
2. Dietterich, T. G., and Bakiri, G. (1995). "Solving Multiclass Learning Problems via Error-Correcting Output Codes." Journal of Artificial Intelligence Research, 2, 263-282. https://arxiv.org/abs/cs/9501101 ↩
3. Crammer, K., and Singer, Y. (2001). "On the Algorithmic Implementation of Multiclass Kernel-based Vector Machines." Journal of Machine Learning Research, 2, 265-292. https://jmlr.csail.mit.edu/papers/volume2/crammer01a/crammer01a.pdf ↩
4. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016). "Rethinking the Inception Architecture for Computer Vision." CVPR. https://arxiv.org/abs/1512.00567 ↩
5. Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollar, P. (2017). "Focal Loss for Dense Object Detection." ICCV. https://arxiv.org/abs/1708.02002 ↩
6. Ridnik, T., Ben-Baruch, E., Noy, A., and Zelnik-Manor, L. (2021). "ImageNet-21K Pretraining for the Masses." NeurIPS Datasets and Benchmarks. https://arxiv.org/abs/2104.10972 ↩
7. Radford, A. et al. (2021). "Learning Transferable Visual Models From Natural Language Supervision" (CLIP). ICML. https://arxiv.org/abs/2103.00020 ↩
8. scikit-learn developers. "cohen_kappa_score." https://scikit-learn.org/stable/modules/generated/sklearn.metrics.cohen_kappa_score.html ↩
9. scikit-learn developers. "f1_score." https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html ↩
10. Wikipedia. "Softmax function." https://en.wikipedia.org/wiki/Softmax_function ↩
11. Russakovsky, O. et al. (2015). "ImageNet Large Scale Visual Recognition Challenge." International Journal of Computer Vision, 115, 211-252. https://arxiv.org/abs/1409.0575 ↩
12. Wikipedia. "CIFAR-10." https://en.wikipedia.org/wiki/CIFAR-10 ↩