Classification (machine learning)

Classification is a supervised learning task in which a system uses labeled examples to assign one or more discrete labels to an input. A classifier may return only a label, such as "spam," or it may return scores or estimated probabilities that a separate decision rule converts into an action. Classification is commonly contrasted with regression, whose target is usually a numeric quantity, and with clustering, which groups observations without training labels.^[1][2]

The basic problem appears in statistics, machine learning, pattern recognition, and decision systems. Inputs can be fixed-length feature vectors, text, images, audio, graphs, or other structured objects. Labels can represent mutually exclusive classes, several attributes at once, an ordered scale, or positions in a hierarchy. The appropriate model and evaluation method depend on that label structure, the costs of different errors, the data-generating process, and the conditions under which the classifier will be used.

Formal problem

In ordinary single-label classification, a training set contains pairs $(x_i,y_i)$, where each input $x_i$ belongs to an input space $\mathcal{X}$ and each target $y_i$ belongs to a finite class set $\mathcal{Y}=\{1,\ldots,K\}$. Learning produces a function from inputs to labels or scores. A hard classifier has the form

$$f:\mathcal{X}\rightarrow\mathcal{Y}.$$

A probabilistic classifier instead estimates quantities such as $p(y=k\mid x)$. Under zero-one loss, choosing the class with the largest true posterior probability minimizes the conditional probability of error. Other applications use a loss or cost matrix. If predicting class $a$ when the true class is $y$ incurs cost $L(a,y)$, the decision rule should choose the action with the smallest posterior expected cost rather than automatically taking the largest score.^[1]

The distinction matters because a model, a probability estimate, and a decision policy are different objects. The same score-producing model can be operated at different thresholds for different use cases. A fraud-screening system, for example, can route high-scoring cases to manual review without asserting that every case above a universal threshold has the same operational meaning.

Training performance alone does not establish that a classifier will work on new data. Statistical learning theory and empirical evaluation generally rely on assumptions connecting the training sample to the future population. Changes in class prevalence, measurement procedures, environments, or user behavior can invalidate an evaluation even when the model was accurate on its original test set.^[2]

Classification, regression, and clustering

Classification and regression are both supervised tasks, but the target and decision problem differ. A regression target is usually quantitative, while a classification target denotes a category. This is not a strict divide between algorithms: a tree, nearest-neighbor method, neural network, or generalized linear model can be adapted to either task. Nor is it determined solely by the final activation function. Some classifiers produce unnormalized margins, some regressors constrain outputs, and ordinal methods occupy part of the boundary between categorical and numeric modeling.

Logistic regression illustrates the naming issue. It models a conditional class probability or log-odds using a regression-like parameterization, but the fitted model is ordinarily used for classification or probability estimation. Conversely, treating arbitrary numeric codes for categories as a continuous response can impose distances and ordering that the labels do not contain.

Clustering differs because its training data do not provide the target class assignments. A cluster is a grouping produced from a similarity criterion or statistical model; it need not coincide with a semantic class. Semi-supervised and self-supervised systems can combine unlabeled data with a supervised classification objective, but the final labeled task remains classification.

Statistical and classical methods

Modern classification developed from statistical decision theory, pattern recognition, and later machine-learning research. In 1936, Ronald Fisher described a linear function of multiple measurements for separating Iris groups, an early and influential discriminant construction.^[3] Current linear discriminant analysis is commonly presented through class-conditional multivariate Gaussian models with a shared covariance matrix; allowing separate covariance matrices yields quadratic decision boundaries.

Linear and probabilistic classifiers

Logistic regression directly parameterizes a conditional class probability. For a binary target,

$$p(y=1\mid x)=\frac{1}{1+\exp[-(w^\top x+b)]}.$$

Regularization can shrink coefficients or induce sparsity, and feature transformations can make the boundary nonlinear in the original measurements. The fitted probabilities are not guaranteed to be calibrated merely because the model uses a logistic function. Misspecification, regularization, sampling design, distribution shift, and the fitting procedure can all affect probability quality.

Naive Bayes instead models the class prior and class-conditional feature distribution, then applies Bayes' rule. The "naive" assumption is conditional independence of features given the class. Gaussian, multinomial, and Bernoulli likelihoods suit different feature representations. The independence assumption can be false while the resulting decision rule remains useful, but performance and probability quality must be measured rather than inferred from the model's simplicity.

Support vector machines

A support vector machine constructs a maximum-margin separator. For linearly separable data, the hard-margin formulation chooses a separating hyperplane with the largest geometric margin. The soft-margin formulation permits violations and trades them against regularization. Cortes and Vapnik's 1995 paper presented the support-vector network and its use with kernels.^[4]

The kernel method replaces inner products with a valid kernel function, allowing a linear separator in an implicit feature space to represent a nonlinear boundary in the original space. Kernel choice and hyperparameters strongly affect the hypothesis class. The raw SVM output is a margin, not inherently a class probability; probability estimates require an additional model or calibration procedure.

Nearest neighbors

The k-nearest neighbors classifier assigns a class using labeled examples close to the query under a selected distance or similarity measure. It performs little or no parametric estimation at training time, although an implementation may normalize features and build a search index. Prediction cost and memory use can be substantial because the reference examples remain part of the fitted system.

Cover and Hart analyzed the nearest-neighbor rule and established asymptotic error bounds under their setting.^[5] Such results do not say that an arbitrary finite-sample k-nearest-neighbor classifier can learn every useful boundary. Distance choice, feature scaling, irrelevant dimensions, sample density, tie handling, and the value of $k$ all influence its behavior.

Decision trees and ensembles

A decision tree recursively partitions the feature space and assigns predictions within terminal regions. Different algorithms use different split criteria and rules for missing or categorical values. It is therefore inaccurate to say that every tree implementation handles every feature type without preprocessing. Deep trees can closely fit training data and have high variance.

Random forests average an ensemble of randomized trees. Breiman's formulation combines bootstrap sampling with random feature selection and analyzes the relation between individual-tree strength and correlation.^[6] Averaging can reduce variance, but the result is still affected by label quality, class imbalance, feature representation, and distribution shift.

Gradient boosting builds an additive model in stages, fitting each new component to improve a differentiable objective. Friedman's gradient-boosting formulation interprets this process as numerical optimization in function space.^[7] XGBoost adds a regularized objective, second-order optimization, sparsity-aware processing, and systems techniques for scalable tree learning.^[8] These papers establish algorithms and reported experiments, not a universal ranking for all tabular datasets.

Generative and discriminative classifiers

The generative-discriminative distinction is about what distribution is modeled. A generative classifier specifies a joint distribution or its components, such as $p(x\mid y)p(y)$; a discriminative classifier models $p(y\mid x)$ or a decision boundary directly. In a particular comparison of naive Bayes and logistic regression, Ng and Jordan proved different asymptotic and sample-size behavior under stated assumptions. Their result does not establish that one generative or discriminative algorithm is universally preferable.^[9]

Generative modeling can support operations beyond assigning a class, such as computing a likelihood under an explicitly modeled feature distribution. Those capabilities depend on the chosen model and its assumptions; not every method called generative provides reliable imputation, synthetic data, or outlier detection. Discriminative models likewise range from calibrated conditional-probability models to margin functions with no inherent probabilistic interpretation.

Task structures

The phrase "classification problem" covers several different output structures.

Scroll sideways for more →

Binary and multiclass classification

Binary classification uses two classes, often called positive and negative for purposes of evaluation. The names do not imply that one class is desirable; they designate which class is used in definitions such as precision and recall. A binary model may output a probability, a distance to a boundary, or another score.

In multiclass classification, the labels are mutually exclusive. A model can estimate all class scores jointly, as multinomial logistic regression and many neural networks do, or reduce the task to binary subproblems. One-versus-rest, one-versus-one, and error-correcting output codes define different decompositions. Their behavior depends on the binary learner, coding scheme, and rule used to combine outputs; reduction is not merely an interchangeable implementation detail.^[10]

Ordinal classification

Ordinal labels have a meaningful order but do not specify equal numerical intervals. Examples include ratings such as poor, fair, and good. Modeling them as unrelated classes discards ordering, while ordinary least-squares regression on arbitrary class numbers adds an interval-scale assumption. McCullagh's cumulative-link framework models cumulative class probabilities through ordered thresholds; the proportional-odds model is one important member of this family.^[11]

Evaluation should also respect the order. Exact accuracy treats every wrong label alike, whereas absolute rank error, squared rank error, or a task-specific cost can distinguish a one-level error from a more distant one. The choice should follow the meaning of the labels rather than the notation used to encode them.

Multilabel classification

In multilabel classification, one observation can have several labels, such as a document tagged with both economics and public policy. A common baseline fits one binary classifier per label, but this "binary relevance" construction does not model dependencies among labels. Other approaches transform the label space, construct classifier chains, rank candidate labels, or learn a joint structured predictor. Multilabel methods also differ in whether they optimize per-label errors, whole-label-set accuracy, or ranking quality.^[12]

Because the number of negative labels can be much larger than the number of positive labels, a single aggregate accuracy can be uninformative. Hamming loss counts label-wise disagreements, while subset accuracy requires the entire predicted set to match. Macro and micro averaging answer different questions when labels have very different frequencies.

Hierarchical and open-set classification

Hierarchical classification uses a tree or directed acyclic graph of labels. A system may predict only leaves, may return labels at several levels, or may stop at an internal node when evidence is insufficient for a more specific decision. Metrics can give partial credit based on taxonomic distance, but such credit is appropriate only when the hierarchy represents the application's notion of similarity.

Closed-set evaluation assumes every test example belongs to a class represented during training. Open-set recognition relaxes that assumption and requires a system to reject or mark some inputs as unknown. Scheirer and colleagues formalized open-space risk and described an open-set version of a support vector classifier. Detecting unfamiliar inputs, however, is not equivalent to discovering or naming the unseen class.^[13]

Neural and pretrained classifiers

Neural classification systems learn representations jointly with the decision function. A network may map an input to class logits and train with cross-entropy, but this is one common design rather than the definition of a classifier. Binary and multilabel heads often use separate logistic outputs; mutually exclusive multiclass heads often use a softmax. Structured tasks can use different output layers and objectives.

Image classification

The ImageNet Large Scale Visual Recognition Challenge helped make large-scale image classification a common benchmark. The official 2012 results report a 15.315 percent top-five test error for the SuperVision entry and 26.172 percent for the next listed entry.^[14] The associated AlexNet paper described a deep convolutional network with 60 million parameters and 650,000 neurons trained on roughly 1.2 million ImageNet images. It reported 15.3 percent top-five test error, compared with 26.2 percent for the second-best challenge entry.^[15] These are results on the challenge's defined task, not a measure of general visual understanding.

Residual networks introduced identity shortcut connections so that stacked layers could learn residual functions. He and colleagues reported that their residual nets made substantially deeper image models easier to optimize and that an ensemble achieved 3.57 percent top-five error on the ILSVRC 2015 test set.^[16] Residual connections became an important design pattern, but later architectures vary considerably and do not all use the original residual block.

The Vision Transformer represents an image as a sequence of embedded patches and applies a Transformer encoder. Dosovitskiy and colleagues reported that, when pretrained on sufficiently large datasets and transferred to several image benchmarks, their models attained results comparable to or better than the convolutional baselines evaluated in the paper while using fewer computational resources for pretraining in those comparisons.^[17] The result is conditional on the paper's data, scale, and training setup; it is not evidence that attention models always outperform convolutional models.

Text classification

Text classifiers can use sparse word or character features, recurrent networks, convolutional networks, or Transformer representations. BERT pretrains a bidirectional Transformer encoder with masked-language-model and next-sentence-prediction objectives, then adds small task-specific output layers for fine-tuning. Its paper reported new results on eleven natural-language-processing tasks under the stated benchmark protocols.^[18]

Encoder fine-tuning is only one option. A pretrained representation can be frozen and supplied to a linear classifier; a generative language model can be prompted to emit a label; or a retrieval system can compare an input embedding with labeled examples. Each choice has different costs, latency, data requirements, and failure modes. Generative output also needs parsing or constrained decoding if only a fixed set of labels is valid.

Zero-shot and few-shot classification

Zero-shot classification uses a model to predict labels for a target task without fitting on labeled examples from that task. CLIP learned paired image and text representations from 400 million image-text pairs collected from the internet. Its paper evaluated zero-shot transfer by comparing an image embedding with text embeddings for candidate class descriptions, and reported zero-shot results across more than 30 computer-vision datasets.^[19] Those results depend on the pretraining data, prompt templates, label descriptions, and relationship between the evaluation set and the learned representation.

Large autoregressive language models can also perform classification from instructions or examples placed in the input context. The GPT-3 paper evaluated zero-shot, one-shot, and few-shot settings without gradient updates to the model.^[20] A prompted model's output is sensitive to choices that may appear incidental. Zhao and colleagues showed that few-shot classification performance could vary substantially with prompt format, demonstration choice, and example order; their contextual-calibration method used content-free inputs to adjust the model's output distribution.^[21]

Prompt-based scores should therefore be validated like other classifier outputs. Label wording can change the effective task, and a model may produce text outside the permitted class set. Zero-shot transfer can reduce the need for task-specific labels, but it does not remove the need for representative evaluation data.

Training objectives

The operational objective is usually a task loss on future observations. Directly minimizing empirical zero-one error is computationally difficult for many model classes and supplies no gradient for ordinary neural-network training, so learning algorithms use surrogate losses.

Cross-entropy and related losses

For a mutually exclusive $K$-class problem, cross-entropy on one labeled example is

$$-\log p(y\mid x),$$

where $p(y\mid x)$ is the probability assigned to the observed class. Averaging this quantity is equivalent to negative conditional log-likelihood for the specified probabilistic model. In binary classification it becomes logistic log loss. Multilabel systems often sum or average a binary log loss for each label, though a joint model may use another objective.

Hinge loss, $\max(0,1-yf(x))$ for binary labels encoded as $-1$ and $+1$, is associated with the soft-margin support vector machine. It encourages a signed margin of at least one rather than fitting a probability. Exponential loss underlies AdaBoost. These losses can produce similar labels while giving scores with different interpretations.

Focal loss multiplies cross-entropy by a factor that reduces the contribution of well-classified examples. Lin and colleagues proposed it for the extreme foreground-background imbalance encountered by one-stage dense object detectors, and evaluated it with RetinaNet.^[22] Applying focal loss to a different classification problem is an additional design choice; the original detector results do not guarantee improvement elsewhere.

Weighting, regularization, and data construction

Class or example weights can represent unequal sampling or error costs, but those purposes should not be conflated. Reweighting to estimate performance in a target population requires knowledge or assumptions about the sampling process. Reweighting to encode utility deliberately changes which errors matter. Either can alter the fitted probability scale.

Regularization constrains model complexity through penalties, architecture, early stopping, data augmentation, or other procedures. Its effect must be assessed on data not used to choose the regularization strength. Data augmentation is valid when the transformation preserves the intended label; a transformation that changes the class introduces mislabeled training examples.

Evaluation

Evaluation begins by defining the population, prediction unit, label process, and decision to be supported. A random row split is inappropriate when rows from the same person, device, document, or time episode can appear on both sides and leak information. Temporal or group-aware splits may better represent deployment. A benchmark result is meaningful only with its dataset, split, preprocessing, metric, and uncertainty.

Confusion matrix and threshold metrics

For a binary task, a confusion matrix counts true positives (TP), false positives (FP), false negatives (FN), and true negatives (TN) at a particular decision threshold.

Scroll sideways for more →

Undefined cases need an explicit convention. Precision, for example, has a zero denominator when a system predicts no positives. Multiclass and multilabel reporting must also state how per-class values are averaged. Macro averaging gives each class or label equal weight; micro averaging pools decisions; support-weighted averaging weights by observed frequency. They can produce markedly different summaries.

Accuracy is appropriate when correct classification is the relevant outcome and the evaluation sample represents the intended population. It can hide poor minority-class performance, but no alternative metric is automatically suitable. F1 ignores true negatives and combines precision with recall; it does not encode every possible cost tradeoff.

Probability quality and calibration

A probabilistic classifier is calibrated when predicted probabilities agree with outcome frequencies under a specified notion of calibration. For binary predictions, one intuitive statement is that events assigned probability near 0.8 occur about 80 percent of the time. Calibration is distinct from discrimination: a model can rank examples well while its numerical probabilities are systematically too high or too low.

Guo and colleagues found that several modern neural networks in their experiments were miscalibrated and examined effects associated with depth, width, weight decay, and batch normalization. They reported temperature scaling as an effective post-hoc method on the datasets studied.^[23] Temperature scaling divides multiclass logits by one fitted positive scalar before softmax. Because the same positive scale preserves the logit order, it does not change top-one predictions, but its calibration benefit must be measured on new data.

Niculescu-Mizil and Caruana compared probability predictions from several supervised learning methods and evaluated Platt scaling and isotonic regression on held-out data.^[24] Platt scaling fits a sigmoid mapping from scores to probabilities. Isotonic regression fits a nondecreasing mapping and is more flexible, which can make it more data-hungry. Calibration data must be separated from the data used to report final performance.

Log loss and the Brier score evaluate probability forecasts rather than a single thresholded label. A strictly proper scoring rule is optimized in expectation by reporting the forecaster's true distribution, with uniqueness in the strictly proper case. Gneiting and Raftery developed the theory and cataloged proper scoring rules.^[25] A lower proper score does not identify whether an error came from calibration, class overlap, or shift, so graphical and class-conditional diagnostics can still be useful.

Ranking metrics

The receiver operating characteristic curve plots true-positive rate against false-positive rate as a threshold varies. ROC area summarizes ranking performance. Under the usual interpretation, it is the probability that a randomly selected positive receives a higher score than a randomly selected negative, with a convention for ties. It does not by itself select an operating threshold or measure probability calibration.^[26]

A precision-recall curve plots precision against recall. Precision depends on class prevalence, so a PR curve from one population cannot be interpreted independently of its sampling ratio. Saito and Rehmsmeier showed why ROC plots can present an overly optimistic visual account on highly imbalanced datasets and argued that precision-recall plots are more informative in that setting.^[27] Both curves still need an operating point when the deployed system must take a concrete action.

Thresholds and decision costs

The threshold 0.5 has no universal status. It follows from a particular binary setup with calibrated posteriors, symmetric error costs, and an action for each class. Different false-positive and false-negative costs imply a different decision boundary, and a review queue or abstention option adds further actions.

Threshold selection must be performed on development data, not the final test set. The selection criterion should reflect deployment utility or constraints, such as a maximum review volume or a minimum sensitivity. If prevalence or costs change after deployment, the selected operating point may need revision even if the score ranking remains stable.

Imbalanced and noisy data

Class imbalance means classes have different frequencies, but rarity alone does not determine the correct remedy. The relevant questions are whether the sample represents deployment, whether error costs are unequal, and whether enough minority examples exist to learn and evaluate the desired behavior.

SMOTE constructs synthetic minority examples by interpolating between a minority example and selected minority neighbors. Chawla and colleagues introduced the method and evaluated combinations of oversampling and undersampling with several classifiers.^[28] Synthetic observations should be generated only inside the training portion of each resampling split. Generating them before the split can place related information in both training and evaluation data.

Broader strategies include class-weighted objectives, cost-sensitive learning, over- or undersampling, ensembles, threshold adjustment, and collection of additional representative data. He and Garcia's survey organizes major challenges and methods for learning from imbalanced data.^[29] No method can recover subgroups that were never observed, and altering the training prevalence can require later correction if probability estimates are intended to describe the deployment population.

Model selection and uncertainty

Hyperparameter search, feature selection, preprocessing choices, threshold tuning, and architecture selection are all parts of model fitting. Repeatedly choosing the best result on the same validation data can overfit that data. Cawley and Talbot showed how model-selection overfitting can create optimistic performance estimates and argued that evaluation must include the selection procedure.^[30]

Nested cross-validation separates an inner model-selection loop from an outer performance-estimation loop. Varma and Simon demonstrated selection bias when the same cross-validation process was used both to tune a classifier and estimate its error, and showed that nested cross-validation reduced that bias in their experiments.^[31] It is not necessary in every fixed-model evaluation, but some independent outer assessment is needed when choices are data-driven.

Comparing many algorithms across many datasets also requires statistical care. Demsar reviewed tests for classifier comparisons and recommended nonparametric procedures, including the Friedman test with suitable post-hoc analysis, for comparisons over multiple datasets.^[32] The experimental unit is the dataset in that setting; treating every cross-validation fold as independent can understate uncertainty.

Distribution shift, rejection, and uncertainty sets

An independent and identically distributed test set estimates performance for its represented distribution. Deployment can introduce covariate shift, label shift, concept change, sensor failure, or previously unseen classes. These phenomena are not interchangeable and may require different diagnostics.

Ovadia and colleagues evaluated predictive uncertainty for neural networks under several dataset shifts and found that uncertainty quality degraded as shift increased, with method-dependent differences.^[33] High softmax confidence is therefore not proof that an input is familiar. Monitoring can track inputs, scores, predicted classes, delayed outcomes, and data-pipeline health, but a statistical alarm must still be investigated to identify its cause.

Selective classification permits a model to abstain on some inputs. Coverage is the fraction on which it predicts, while selective risk measures error among accepted predictions. Geifman and El-Yaniv proposed a method for controlling the risk of a deep classifier while maximizing coverage under their sampling assumptions.^[34] Abstention is useful only if the deferred cases have a safe destination, such as human review or another system.

Conformal prediction can turn classifier scores into prediction sets. Under exchangeability and a correctly implemented calibration procedure, standard split-conformal methods provide a finite-sample marginal coverage guarantee. The guarantee concerns repeated observations from the assumed data-generating process, not conditional accuracy for every class or subgroup. Angelopoulos and Bates explain the assumptions, algorithms, and limitations of conformal uncertainty sets and intervals.^[35]

Open-set recognition, out-of-distribution detection, selective prediction, and conformal prediction overlap but answer different questions. An open-set method seeks to identify inputs outside known classes; selective prediction trades coverage for risk; conformal prediction targets a coverage property for a set-valued output. A deployed system may use more than one.

Fairness, robustness, and governance

Classification performance can differ across groups because of sampling, measurement, label definitions, feature availability, historical decisions, or model behavior. Aggregate accuracy can conceal those differences. The appropriate assessment depends on the decision context and on which quantities are legally and ethically relevant.

Hardt, Price, and Srebro formulated equality of opportunity using group-conditional error rates and studied post-processing of a learned predictor.^[36] Fairness criteria can conflict when base rates differ, and matching one metric does not establish that a system is fair. Group definitions, intersectional sample sizes, label validity, accessibility, and the consequences of false positives and false negatives must also be examined.

Classifiers can be sensitive to deliberate perturbations. Goodfellow, Shlens, and Szegedy described adversarial examples and proposed the fast gradient sign method in the context of neural networks.^[37] Robustness to that attack does not imply robustness to every adversary, natural corruption, or data-pipeline failure. Threat modeling should identify the attacker's capabilities and the system assets at risk.

NIST Special Publication 1270 describes sources of harmful bias across the artificial-intelligence lifecycle and frames bias management as a sociotechnical responsibility, not only a model-metric exercise.^[38] The NIST AI Risk Management Framework similarly organizes voluntary risk management around the functions Govern, Map, Measure, and Manage.^[39] These frameworks do not certify a classifier. They provide processes for identifying context, documenting evidence, assigning responsibilities, and revisiting risk over time.

Operational governance commonly includes:

• a documented target and label policy;
• data provenance, inclusion criteria, and known limitations;
• versioned preprocessing, features, model, calibration, and threshold;
• evaluation by relevant classes, groups, sites, and time periods;
• human-review and appeal procedures where decisions affect people;
• monitoring for input, output, and outcome changes;
• rollback criteria and incident response;
• periodic reassessment when the environment or purpose changes.

Choosing a method

No classifier is best for every problem. More generally, Wolpert and Macready's no-free-lunch results for optimization show that, when performance is averaged uniformly over all possible objective functions in their formal setting, algorithms have equal average performance.^[40] The theorem is not a direct comparison of classification methods and does not say that all methods perform equally on a specified real-world distribution. A preference for a classifier must come from assumptions or evidence about the problem at hand.

Label quality is part of that evidence. Label noise can arise from annotation disagreement, ambiguous definitions, measurement error, weak supervision, or changes in policy. Random symmetric flips are only one model of that process. Frenay and Verleysen surveyed terminology, sources, consequences, and learning methods for classification with label noise.^[41] Before applying a noise-robust loss, an audit should determine whether disagreements reveal poor annotation, a genuinely ambiguous task, or a target definition that needs revision.

A useful comparison starts with the deployment requirement rather than a leaderboard:

Scroll sideways for more →

Simple baselines are valuable because they reveal whether extra complexity improves the relevant outcome. Depending on the representation and sample size, these may include a majority-class rule, regularized logistic regression, naive Bayes, a shallow tree, nearest neighbors, or a linear support vector machine. More complex ensembles and neural models should be compared using the same splits, preprocessing boundaries, selection budget, and decision criteria.

The final model is only one component of a classification system. Label creation, sampling, feature extraction, training, calibration, thresholding, human escalation, and monitoring can each dominate the observed error. Reproducible evaluation therefore records the complete pipeline rather than only the classifier name.

Applications

Classification is used when a discrete assignment supports a defined task. Examples include:

• assigning documents to topics or routing categories;
• identifying objects or scene categories in images;
• recognizing an intended command from speech or text;
• screening transactions for further fraud investigation;
• triaging records for expert review;
• tagging products, media, or scientific observations with multiple attributes;
• categorizing equipment states from sensor measurements.

These examples do not imply that one algorithm, metric, or threshold is appropriate across domains. A medical screening label, an email folder, and a manufacturing defect code have different evidence standards and consequences. In high-impact uses, the classifier's output should be treated as evidence within a governed decision process, not as a self-validating fact.

See also

• Supervised learning
• Binary classification
• Multi-class classification
• Multilabel classification
• Regression
• Clustering
• Confusion matrix
• Logistic regression
• Naive Bayes
• Support vector machine
• K-nearest neighbors
• Decision tree
• Random forest
• Gradient boosting
• Calibration

References

1. ^Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer. microsoft.com/...tern-recognition-machine-learning
2. ^Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, second edition. Springer. hastie.su.domains/...main
3. ^Fisher, R. A. (1936). The Use of Multiple Measurements in Taxonomic Problems. Annals of Eugenics, 7(2), 179-188. repository.rothamsted.ac.uk/...-taxonomic-problems
4. ^Cortes, C., and Vapnik, V. (1995). Support-Vector Networks. Machine Learning, 20, 273-297. doi.org/...BF00994018
5. ^Cover, T. M., and Hart, P. E. (1967). Nearest Neighbor Pattern Classification. IEEE Transactions on Information Theory, 13(1), 21-27. doi.org/...TIT.1967.1053964
6. ^Breiman, L. (2001). Random Forests. Machine Learning, 45, 5-32. doi.org/...A:1010933404324
7. ^Friedman, J. H. (2001). Greedy Function Approximation: A Gradient Boosting Machine. The Annals of Statistics, 29(5), 1189-1232. doi.org/...1013203451
8. ^Chen, T., and Guestrin, C. (2016). XGBoost: A Scalable Tree Boosting System. Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 785-794. arxiv.org/...1603.02754
9. ^Ng, A. Y., and Jordan, M. I. (2002). On Discriminative vs. Generative Classifiers: A Comparison of Logistic Regression and Naive Bayes. Advances in Neural Information Processing Systems 14. proceedings.neurips.cc/...d6be6c91c4f6c4e-Abstract
10. ^Allwein, E. L., Schapire, R. E., and Singer, Y. (2000). Reducing Multiclass to Binary: A Unifying Approach for Margin Classifiers. Journal of Machine Learning Research, 1, 113-141. jmlr.org/...allwein00a
11. ^McCullagh, P. (1980). Regression Models for Ordinal Data. Journal of the Royal Statistical Society: Series B (Methodological), 42(2), 109-142. people.csail.mit.edu/...mccullagh-ordinal-80.pdf
12. ^Tsoumakas, G., and Katakis, I. (2007). Multi-Label Classification: An Overview. International Journal of Data Warehousing and Mining, 3(3), 1-13. people.iee.ihu.gr/...cation%3A%20An%20Overview.pdf
13. ^Scheirer, W. J., de Rezende Rocha, A., Sapkota, A., and Boult, T. E. (2013). Toward Open Set Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(7), 1757-1772. wjscheirer.com/...wjs_tpami2013_openset.pdf
14. ^ImageNet. (2012). ImageNet Large Scale Visual Recognition Challenge 2012 Results. image-net.org/...results
15. ^Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). ImageNet Classification with Deep Convolutional Neural Networks. Advances in Neural Information Processing Systems 25. proceedings.neurips.cc/...8436e924a68c45b-Abstract
16. ^He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep Residual Learning for Image Recognition. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 770-778. openaccess.thecvf.com/..._Learning_CVPR_2016_paper
17. ^Dosovitskiy, A., et al. (2021). An Image Is Worth 16x16 Words: Transformers for Image Recognition at Scale. International Conference on Learning Representations. openreview.net/forum
18. ^Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of NAACL-HLT 2019, 4171-4186. aclanthology.org/N19-1423
19. ^Radford, A., et al. (2021). Learning Transferable Visual Models From Natural Language Supervision. Proceedings of the 38th International Conference on Machine Learning, 8748-8763. proceedings.mlr.press/...radford21a
20. ^Brown, T. B., et al. (2020). Language Models Are Few-Shot Learners. Advances in Neural Information Processing Systems 33, 1877-1901. proceedings.neurips.cc/...18bfb8ac142f64a-Abstract
21. ^Zhao, Z., Wallace, E., Feng, S., Klein, D., and Singh, S. (2021). Calibrate Before Use: Improving Few-shot Performance of Language Models. Proceedings of the 38th International Conference on Machine Learning, 12697-12706. proceedings.mlr.press/...zhao21c
22. ^Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollar, P. (2017). Focal Loss for Dense Object Detection. Proceedings of the IEEE International Conference on Computer Vision, 2980-2988. openaccess.thecvf.com/..._Loss_for_ICCV_2017_paper
23. ^Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). On Calibration of Modern Neural Networks. Proceedings of the 34th International Conference on Machine Learning, 1321-1330. proceedings.mlr.press/...guo17a
24. ^Niculescu-Mizil, A., and Caruana, R. (2005). Predicting Good Probabilities With Supervised Learning. Proceedings of the 22nd International Conference on Machine Learning, 625-632. icml.cc/...Probabilities_NiculescuMizilCaruana.pdf
25. ^Gneiting, T., and Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477), 359-378. sites.stat.washington.edu/...Gneiting2007jasa.pdf
26. ^Fawcett, T. (2006). An Introduction to ROC Analysis. Pattern Recognition Letters, 27(8), 861-874. doi.org/...j.patrec.2005.10.010
27. ^Saito, T., and Rehmsmeier, M. (2015). The Precision-Recall Plot Is More Informative than the ROC Plot When Evaluating Binary Classifiers on Imbalanced Datasets. PLOS ONE, 10(3), e0118432. doi.org/...journal.pone.0118432
28. ^Chawla, N. V., Bowyer, K. W., Hall, L. O., and Kegelmeyer, W. P. (2002). SMOTE: Synthetic Minority Over-sampling Technique. Journal of Artificial Intelligence Research, 16, 321-357. jair.org/...10302
29. ^He, H., and Garcia, E. A. (2009). Learning From Imbalanced Data. IEEE Transactions on Knowledge and Data Engineering, 21(9), 1263-1284. doi.org/...TKDE.2008.239
30. ^Cawley, G. C., and Talbot, N. L. C. (2010). On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation. Journal of Machine Learning Research, 11, 2079-2107. jmlr.org/...cawley10a
31. ^Varma, S., and Simon, R. (2006). Bias in Error Estimation When Using Cross-Validation for Model Selection. BMC Bioinformatics, 7, 91. doi.org/...1471-2105-7-91
32. ^Demsar, J. (2006). Statistical Comparisons of Classifiers over Multiple Data Sets. Journal of Machine Learning Research, 7, 1-30. jmlr.org/...demsar06a
33. ^Ovadia, Y., et al. (2019). Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. Advances in Neural Information Processing Systems 32. proceedings.neurips.cc/...371888657d2eb1d-Abstract
34. ^Geifman, Y., and El-Yaniv, R. (2017). Selective Classification for Deep Neural Networks. Advances in Neural Information Processing Systems 30. proceedings.neurips.cc/...b7e46540e2b0cf1-Abstract
35. ^Angelopoulos, A. N., and Bates, S. (2023). Conformal Prediction: A Gentle Introduction. Foundations and Trends in Machine Learning, 16(4), 494-591. arxiv.org/...2107.07511
36. ^Hardt, M., Price, E., and Srebro, N. (2016). Equality of Opportunity in Supervised Learning. Advances in Neural Information Processing Systems 29. proceedings.neurips.cc/...7384ba499518b38-Abstract
37. ^Goodfellow, I. J., Shlens, J., and Szegedy, C. (2015). Explaining and Harnessing Adversarial Examples. International Conference on Learning Representations. arxiv.org/...1412.6572
38. ^Schwartz, R., et al. (2022). Towards a Standard for Identifying and Managing Bias in Artificial Intelligence. NIST Special Publication 1270. doi.org/...NIST.SP.1270
39. ^Tabassi, E. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. doi.org/...NIST.AI.100-1
40. ^Wolpert, D. H., and Macready, W. G. (1997). No Free Lunch Theorems for Optimization. IEEE Transactions on Evolutionary Computation, 1(1), 67-82. doi.org/...4235.585893
41. ^Frenay, B., and Verleysen, M. (2014). Classification in the Presence of Label Noise: A Survey. IEEE Transactions on Neural Networks and Learning Systems, 25(5), 845-869. pubmed.ncbi.nlm.nih.gov/24808033