Supervised Learning

Supervised learning is a machine learning paradigm in which a model is fitted to examples that pair an input with an observed target. The trained model is then used to predict targets for inputs that were not used to fit it. A target may be a category, a numerical quantity, an ordered label, a ranking, or a more structured object. Classification and regression are the two standard cases.^[1][2]

The labels provide a learning signal, but they are not necessarily error-free or complete descriptions of reality. They may come from measurements, records, human annotation, or another data-generating process. A supervised model therefore learns a relationship present in its training examples. Whether that relationship remains useful depends on the data, the evaluation design, and the conditions under which the model is deployed.^[1][17]

Formal definition

Let the input space be X, the target space be Y, and a training sample contain n pairs:

$$S = \{(x_i, y_i)\}_{i=1}^{n}.$$

A learning algorithm uses S to select a function f from a hypothesis class H. The function maps an input in X to a prediction in Y, or to parameters of a predictive distribution over Y. The hypothesis class may be a set of linear functions, decision trees, kernel functions, neural networks, or another family of predictors.^[1][2]

A loss function L(y, f(x)) assigns a cost to a prediction. The expected risk under a target distribution P(X, Y) is:

$$R(f) = \mathbb{E}_{(X,Y) \sim P}[L(Y, f(X))].$$

The distribution and its expected risk are generally unknown. Empirical risk minimization instead minimizes the average loss on the observed sample:

$$\hat{R}_S(f) = \frac{1}{n}\sum_{i=1}^{n} L(y_i, f(x_i)).$$

Practical objectives often add a penalty that constrains model complexity:

$$\hat{f} = \arg\min_{f \in \mathcal{H}}\left(\hat{R}_S(f) + \lambda\Omega(f)\right).$$

Here, lambda controls the strength of the penalty. The training loss, penalty, hypothesis class, and optimization procedure together define what can be learned from a finite sample.^[1][3][4]

Generalization

Generalization is performance on relevant examples outside the training sample. A low training loss does not establish that a model will generalize. The training examples may be unrepresentative, the model may fit sample-specific noise, or the evaluation data may have influenced preprocessing or model selection.^[1][3][8]

Many theoretical results assume that training and future examples are independent draws from the same distribution. That assumption makes risk estimation tractable, but it does not hold automatically. Repeated observations from one person, measurements from the same device, neighboring spatial samples, and time-ordered records may be dependent. Deployment data may also differ from the data used during development.^[6][18]

Types of supervised task

The target's structure determines the prediction task and narrows the choice of losses and metrics.^[1][7]

Scroll sideways for more →

The boundaries are sometimes chosen for operational reasons. An age may be modeled as a continuous value or placed into age bands. A continuous risk score may later be thresholded into a binary action. These choices change the objective and the meaning of an error; they are not merely formatting decisions.^[1][7]

Classification

Binary classification has two classes. Multiclass classification has more than two mutually exclusive classes. Multilabel classification permits several labels for one example. A classifier may return only a label, but many methods first produce a score or estimated probability and then apply a decision rule.^[1][7]

A default threshold such as 0.5 is not generally optimal. The selected threshold depends on the output's interpretation, the relative consequences of false positives and false negatives, and the class distribution in the intended setting. Threshold selection belongs inside the validation process, not on the final test sample.^[7][9]

Regression

Regression predicts a quantitative target. Least-squares regression estimates a conditional mean under its usual interpretation, while absolute-error and quantile losses target different summaries of the conditional distribution. Two regression models can therefore make different, internally consistent predictions because they optimize different losses.^[1][21]

Regression is not limited to a single scalar. Multi-output regression predicts several quantities together. Probabilistic regression estimates a distribution or distribution parameters rather than only a point estimate. Evaluation should match the output being claimed, such as point accuracy, interval coverage, or probabilistic fit.^[7][21]

Data and labels

Each supervised example contains features and a target. A feature vector can include numerical measurements, categorical variables, text, images, audio, graphs, or learned representations. The target may be measured directly, derived from later events, or assigned by annotators.^[1][2]

The word "label" can imply a definitive answer, but observed targets may contain uncertainty. Medical diagnoses may change with additional testing. Human annotators can disagree. A business outcome may be censored or recorded through a process that excludes some cases. A model learns from the observed target, including systematic errors in how that target was created.^[17]

Dataset documentation can record why data was collected, what population it describes, how instances and labels were obtained, known exclusions, recommended uses, and maintenance practices. This information helps a later user judge whether the sample and target fit a proposed prediction task.^[19]

Sampling and representativeness

The population represented in the training sample matters as much as sample size. A large convenience sample can still omit important groups, conditions, devices, or time periods. Sampling decisions can also change the apparent class prevalence, which affects metrics such as precision and the interpretation of predicted probabilities.^[18][20]

Training, validation, and test data should reflect the unit on which the system must generalize. If multiple rows belong to the same patient, machine, household, document, or event, random row-level splitting can place closely related information on both sides of a split. Group-aware splitting keeps the intended unit separate. Time-dependent tasks usually require chronological evaluation so that the model is not trained on information from the future.^[6][8]

Preprocessing and feature construction

Common preprocessing steps include imputation, numerical scaling, categorical encoding, text or image transformation, and feature engineering. The parameters of a transformation are part of the fitted model. Means used for scaling, vocabularies, category statistics, selected features, and dimensionality-reduction components must be learned only from the training portion of each split.^[8]

This rule also applies during cross-validation. Each fold must refit the complete pipeline on its training folds and apply the fitted transformations to its validation fold. Fitting a transformation once on the full dataset before cross-validation leaks information and can make the estimate optimistic.^[1][8]

Training and model selection

A supervised-learning workflow separates parameter fitting, model selection, and final assessment. The exact split design depends on the amount of data and its dependence structure; there is no universally correct percentage allocation.^[1][6]

Scroll sideways for more →

Holdout evaluation

In a holdout design, development data is split into training and validation portions, while a test portion remains separate. The training portion fits model parameters. Validation results guide choices such as model family, hyperparameters, features, and decision thresholds. The test set is evaluated after those choices have been fixed.^[4][6]

The test result answers a question about the test sampling process. It does not prove that performance will remain unchanged under a different population, a later time period, or a new measurement system. Repeatedly choosing changes based on test results makes the test set part of development and weakens its role as independent evidence.^[1][6]

Cross-validation

In k-fold cross-validation, the development sample is divided into k folds. The procedure fits on k - 1 folds and evaluates on the remaining fold, repeating until each fold has been held out. The fold results are aggregated to estimate the performance of the whole fitted procedure.^[1][6]

The split iterator must match the data. Stratified splits preserve approximate class proportions. Grouped splits keep related observations together. Time-series splits preserve temporal order. Random folds are unsuitable when exchangeability between observations is implausible.^[6]

When cross-validation selects hyperparameters and also estimates performance, the same fold results have influenced the selected model. Nested cross-validation can separate an inner selection loop from an outer assessment loop when an independent test set is unavailable. It costs more computation and does not fix a mismatch between the sampled data and the deployment population.^[6]

Fitting and optimization

Some supervised algorithms have closed-form or specialized fitting procedures. Many models instead use iterative numerical optimization. In neural networks, backpropagation computes derivatives of the objective, and an optimizer such as stochastic gradient descent updates the parameters.^[10]

Optimization error, training loss, and generalization error are different. An optimizer can fail to find a low-loss solution. It can find a low training loss that does not generalize. It can also optimize a surrogate loss whose relationship to the final decision metric depends on thresholding or calibration.^[1][7]

Algorithm families

Supervised learning is a problem setting, not one algorithm. The choice of method depends on the target, data representation, sample size, computational limits, interpretability needs, and evaluation protocol.^[1][5]

Scroll sideways for more →

No model family dominates every supervised problem. Comparing candidates under a shared, leakage-free protocol is more reliable than choosing from a generic ranking. Simple baselines are useful because they reveal whether added complexity produces a reproducible improvement.^[1][5]

Deep supervised learning and transfer

Deep learning combines supervised objectives with multilayer neural networks. Convolutional networks, recurrent networks, and transformers can all be trained with labeled examples. Their architecture does not determine the learning paradigm: the same architecture may be trained with supervised labels, self-supervised objectives, reinforcement signals, or a mixture.^[10][21]

Transfer learning separates representation learning from a target task. A model may first be pretrained on a different dataset or objective, then fitted or fine-tuned with labeled examples for the target task. The final stage is supervised when it uses input-target pairs, even if the earlier pretraining stage was self-supervised.^[21]

Transfer does not guarantee improvement. A pretrained representation may omit information needed for the target, encode unwanted correlations, or mismatch the new population. Evaluation must still use data and splits that reflect the target use.^[18][20]

Loss functions

A loss function gives the learner a numerical objective. The loss used for fitting need not be the same statistic reported to readers or the same cost incurred by a real decision.^[1][7]

Classification losses

Scroll sideways for more →

Cross-entropy does not by itself guarantee calibrated probabilities on new data. Calibration is an empirical property of predictions under a defined distribution and should be evaluated separately.^[9][16]

Regression losses

Scroll sideways for more →

Changing the loss changes the estimand. A model trained for the conditional mean is not automatically an estimator of the conditional median or a calibrated prediction interval.^[1][21]

Evaluation

Evaluation begins with the decision being supported. The metric should reflect the target, important error types, class prevalence, and whether the output is a label, score, probability, or numerical estimate.^[7]

Classification metrics

For binary classification, let TP and TN denote correctly predicted positive and negative cases, and FP and FN denote the two error types.

Scroll sideways for more →

Accuracy can be uninformative when one class dominates or when error costs differ. Precision changes with class prevalence, so a value measured in an artificially balanced test set may not transfer to deployment prevalence. ROC AUC, precision-recall summaries, and thresholded metrics answer different questions; none is a universal replacement for the others.^[7]

Multiclass and multilabel results require an averaging convention. Macro averaging gives each class equal weight. Micro averaging pools decisions across classes. Weighted averaging uses class support. The convention should be reported because the same per-class results can produce different aggregate scores.^[7]

Regression metrics

Scroll sideways for more →

Percentage errors need special handling when true values can be zero or close to zero. Aggregated errors can also hide poor performance in a region that matters operationally. Residual plots and subgroup or range-specific results complement a single summary.^[7]

Calibration and uncertainty

A probabilistic classifier is calibrated when, over the evaluated distribution, events assigned a probability near p occur at a frequency near p. A model can rank cases well while giving overconfident or underconfident probabilities. Reliability diagrams and calibration statistics inspect this property.^[9][16]

Calibration procedures such as sigmoid, isotonic, or temperature scaling need data not used to fit the base model. They can improve calibration on data like the calibration sample, but calibration can deteriorate under distribution shift. A probability estimate is therefore conditional on the modeling and evaluation setting, not a permanent property of a model.^[9][16][18]

Overfitting, underfitting, and regularization

Overfitting occurs when a fitted procedure captures sample-specific variation that does not improve, and may harm, performance on relevant new data. Underfitting occurs when the model or training procedure cannot capture enough of the predictive relationship even on the development data.^[1][3]

For squared-error regression under a standard additive-noise formulation, expected prediction error can be decomposed into irreducible noise, squared bias, and variance. This decomposition explains how a flexible fitting procedure may reduce systematic error while becoming more sensitive to which sample it receives. The exact decomposition and interpretation do not transfer unchanged to every loss and task.^[1][3]

Regularization changes the model or fitting procedure to control generalization rather than only training fit.^[1][4]

Scroll sideways for more →

More data can reduce estimation variance when it represents the target population and its labels are informative. More rows do not correct a systematically wrong target, leakage, or a sampling process that omits the cases of interest.^[17][19]

Failure modes

Data leakage

Data leakage occurs when model development uses information that would not be available at prediction time or that belongs to the evaluation sample. Examples include fitting preprocessing on all rows, selecting features before the split, using post-outcome variables, and tuning repeatedly against the test set. Leakage usually produces an optimistic evaluation rather than a better deployable model.^[8]

Label noise and ambiguity

Observed targets may be corrupted, incomplete, or ambiguous. Random label errors can reduce effective sample information. Systematic errors can teach a model the wrong relationship. Multiple annotators may disagree because the task lacks a single objective label, not because one annotator made a simple mistake.^[17]

Label cleaning should preserve provenance. Removing difficult examples solely because a current model disagrees with them can erase valid minority cases or reinforce that model's existing errors. Independent review, adjudication rules, and uncertainty-aware targets are alternatives when the task supports them.^[17][19]

Class imbalance

When classes are imbalanced, a model can achieve high accuracy while failing on the minority class. Remedies may include changing the sampling scheme, using class or example weights, adjusting the decision threshold, or optimizing a cost-sensitive objective. Each intervention changes either the fitted distribution, the objective, or the decision rule, so evaluation should be performed under the intended operating conditions.^[7]

Distribution shift

Distribution shift means that the development and target distributions differ. Under covariate shift, the input distribution changes while the conditional target rule P(Y | X) is assumed stable. Other shifts can change class prevalence or the conditional relationship itself.^[18]

No validation protocol can guarantee performance under arbitrary future change. Temporal and external validation can test specified shifts. Monitoring can detect changes in inputs, predictions, labels, or performance after deployment, but choosing a response requires assumptions about what changed and which new data is trustworthy.^[18][20]

Spurious prediction and causality

A supervised model can use any stable association that reduces its objective, including proxies and artifacts. A classifier may rely on a scanner mark, documentation practice, or background pattern rather than the intended concept. Performance may collapse when that association changes.^[18][19]

Predictive fit alone does not identify a causal effect. A model trained on observational input-target pairs estimates associations under its data and assumptions. Questions about what would happen under an intervention require a causal design or additional assumptions beyond ordinary supervised learning.^[21]

Bias and unequal performance

Training data can encode historical decisions, unequal measurement quality, or underrepresentation. Aggregate metrics may conceal different error rates across groups. NIST distinguishes systemic, computational and statistical, and human-cognitive sources of bias, which means a class-balancing operation cannot address every source.^[20]

Evaluation can report subgroup performance and uncertainty where groups are meaningful and sample sizes permit. The choice of target, error cost, fairness criterion, and response to disparities is a sociotechnical decision, not something the learning algorithm resolves automatically.^[20]

Relationship to other learning paradigms

The same model architecture can participate in several learning paradigms. The distinction comes from the source of the training signal and the problem being solved.^[21]

Scroll sideways for more →

A self-supervised model may later be fine-tuned with supervised labels. Reinforcement-learning systems may contain supervised components, such as a perception model or a value target generated from experience. These combinations do not erase the distinction between their training signals.^[21]

Historical development

Supervised learning developed through overlapping work in statistics, pattern recognition, learning theory, and neural computation. The following milestones concern methods or formalizations directly tied to learning from examples.

Scroll sideways for more →

These milestones did not replace earlier statistical methods. Linear models, trees, kernels, ensembles, and neural networks remain alternatives whose suitability depends on the prediction problem and evidence from a valid evaluation.^[1][5]

Applications

Supervised learning applies when historical or experimentally collected examples connect available inputs to a target of interest. The model's role is prediction; a domain-specific process must determine how predictions are used and reviewed.^[1][20]

Scroll sideways for more →

A model can be accurate for one narrowly defined target while being unsuitable for a broader decision. For example, predicting a recorded action is not the same as predicting need, benefit, or causal response. Target definition and deployment governance remain part of the system design.^[19][20]

Software

Scikit-learn provides a common estimator, preprocessing, pipeline, model-selection, and metric interface for many classical supervised methods. Its pipeline tools are designed to keep learned transformations inside the fitted procedure, which helps prevent leakage during validation.^[5][8]

A library does not choose the target, split strategy, metric, or deployment boundary. Those decisions determine what its reported score means.^[1][6][7][8]

See also

• Machine learning
• Training set
• Test set
• Cross-validation
• Overfitting
• Regularization
• Semi-supervised learning
• Self-supervised learning
• Unsupervised learning
• Reinforcement learning

References

1. ^Hastie, T., Tibshirani, R., and Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed., corrected 12th printing (2017). Springer. Official author-hosted edition.
2. ^Ng, A., and Ma, T. "Supervised Learning, Discriminative Algorithms." Stanford CS229 lecture notes. PDF.
3. ^Ng, A. "Learning Theory." Stanford CS229 lecture notes. PDF.
4. ^Ng, A. "Regularization and Model Selection." Stanford CS229 lecture notes. PDF.
5. ^Scikit-learn developers. "Supervised learning." Scikit-learn User Guide. Documentation.
6. ^Scikit-learn developers. "Cross-validation: evaluating estimator performance." Scikit-learn User Guide. Documentation.
7. ^Scikit-learn developers. "Metrics and scoring: quantifying the quality of predictions." Scikit-learn User Guide. Documentation.
8. ^Scikit-learn developers. "Common pitfalls and recommended practices." Scikit-learn User Guide. Documentation.
9. ^Scikit-learn developers. "Probability calibration." Scikit-learn User Guide. Documentation.
10. ^Rumelhart, D. E., Hinton, G. E., and Williams, R. J. (1986). "Learning representations by back-propagating errors." Nature, 323, 533-536. DOI.
11. ^Valiant, L. G. (1984). "A Theory of the Learnable." Communications of the ACM, 27(11), 1134-1142. DOI.
12. ^Vapnik, V. N., and Chervonenkis, A. Y. (1971). "On the Uniform Convergence of Relative Frequencies of Events to Their Probabilities." Theory of Probability and Its Applications, 16(2), 264-280. DOI.
13. ^Cortes, C., and Vapnik, V. (1995). "Support-vector networks." Machine Learning, 20, 273-297. DOI.
14. ^Breiman, L. (2001). "Random Forests." Machine Learning, 45, 5-32. DOI.
15. ^Friedman, J. H. (2001). "Greedy Function Approximation: A Gradient Boosting Machine." The Annals of Statistics, 29(5), 1189-1232. DOI.
16. ^Guo, C., Pleiss, G., Sun, Y., and Weinberger, K. Q. (2017). "On Calibration of Modern Neural Networks." Proceedings of Machine Learning Research, 70, 1321-1330. Paper and PDF.
17. ^van Rooyen, B., and Williamson, R. C. (2018). "A Theory of Learning with Corrupted Labels." Journal of Machine Learning Research, 18(228), 1-50. Paper and PDF.
18. ^Sugiyama, M. (2006). "Supervised Learning under Covariate Shift." Journal of the Japanese Neural Network Society, 13(3), 111-118. DOI and PDF.
19. ^Gebru, T., Morgenstern, J., Vecchione, B., Vaughan, J. W., Wallach, H., Daume III, H., and Crawford, K. (2021). "Datasheets for Datasets." Communications of the ACM, 64(12), 86-92. DOI.
20. ^Tabassi, E. (2023). Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST AI 100-1. National Institute of Standards and Technology. DOI and PDF.
21. ^Murphy, K. P. (2022). Probabilistic Machine Learning: An Introduction. MIT Press. Official book page.
22. ^Rosenblatt, F. (1958). "The Perceptron: A Probabilistic Model for Information Storage and Organization in the Brain." Psychological Review, 65(6), 386-408. DOI.