Machine learning terms/Fundamentals

See also: Machine learning terms

What are the fundamental terms in machine learning?

Machine learning (ML) is the branch of artificial intelligence concerned with building systems that learn patterns from data rather than following explicitly programmed rules. The field draws on statistics, optimization theory, and computer science, and rests on a small vocabulary of foundational terms: a model (a parameterized function family), features and labels (the inputs and target outputs), a loss function (a measure of prediction error), and an optimizer such as gradient descent that adjusts the parameters to reduce that loss. A canonical formal definition comes from Tom Mitchell (1997): "A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with experience E."^[1]

The term "machine learning" itself was coined by IBM researcher Arthur Samuel in 1959, in his paper "Some Studies in Machine Learning Using the Game of Checkers," published in the IBM Journal of Research and Development.^[25] Samuel framed the goal of the field through his checkers program, writing that "a computer can be programmed so that it will learn to play a better game of checkers than can be played by the person who wrote the program."^[25] The broader paraphrase often attributed to him, that machine learning is the field of study that gives computers the ability to learn without being explicitly programmed, captures the same idea but is a later reformulation rather than a verbatim line from the 1959 paper.

This page collects the foundational vocabulary that underpins almost every ML system, from a simple linear regression baseline to a billion-parameter neural network. Each term is linked to its own dedicated wiki article. The fundamentals chapter is the entry point for readers working through the Machine learning terms glossary, which is grouped by topic into chapters such as fundamentals, neural networks and deep learning, language evaluation, and advanced topics.

A practical ML system has three ingredients: a model (a parameterized function family), a loss function (a measure of how far predictions are from ground truth), and an optimization procedure (a way to adjust the parameters so that the loss decreases). The remainder of this article walks through the categories, components, and metrics that make this loop work in practice.

What are the main categories of machine learning?

ML problems are usually grouped by the kind of supervision signal the training set provides. The five main categories are summarized below.

Scroll sideways for more →

What is supervised learning?

In supervised machine learning the dataset consists of pairs of inputs and target outputs. The learner fits a function that maps inputs to outputs and is evaluated on held-out examples. Most production ML systems, including spam filters, fraud detection, ad ranking, and medical image triage, are supervised.

What is unsupervised learning?

Unsupervised machine learning discovers structure in unlabeled data. Clustering groups similar items, dimensionality reduction projects high-dimensional data into a lower-dimensional space for visualization or compression, and density estimation models the probability distribution of the inputs.

What is the difference between semi-supervised and self-supervised learning?

Semi-supervised learning is useful when labels are expensive but raw data is plentiful. Self-supervised learning has become the dominant paradigm for foundation models: the model is pretrained on large unlabeled corpora using a pretext task (predicting masked tokens, the next token, or the relative position of image patches) and is then fine-tuned on smaller supervised datasets.

What is reinforcement learning?

Reinforcement learning (RL) trains an agent to take actions in an environment so as to maximize cumulative reward. The training signal is sparse and delayed, which makes the credit assignment problem central.^[6] Notable RL systems include DeepMind's AlphaGo and AlphaZero, OpenAI Five, and the reinforcement learning from human feedback (RLHF) stage used to align large language models.

What is the difference between regression and classification?

Within supervised learning, the two main task types are regression and classification.

Scroll sideways for more →

Classification models often output probabilities via a softmax (multi-class) or sigmoid (binary) head, and a classification threshold converts the probability into a discrete decision.

What are bias, variance, overfitting, and generalization?

The central goal of ML is generalization: producing a model that performs well on data it has not seen. Two failure modes block this goal.

• Underfitting occurs when the model is too simple to capture the structure in the data. Both training error and test error are high. Remedies include using a richer model, adding features, or training longer.
• Overfitting occurs when the model memorizes idiosyncrasies of the training set. Training error is low but test error is high. Remedies include collecting more data, simplifying the model, or applying regularization.

What is the bias-variance trade-off?

The expected squared error of a regression estimator can be decomposed into three terms: a bias term (how far the average prediction is from the truth), a variance term (how much predictions fluctuate across training sets), and irreducible noise.^[2] High-bias models (like a linear model on a curved relationship) underfit; high-variance models (like a deep tree on a small dataset) overfit. Choosing model capacity and regularization is largely the art of balancing these two sources of error.

What does a generalization curve show?

A generalization curve plots training loss and validation loss against training time or model capacity. As capacity grows, training loss falls monotonically while validation loss typically falls and then rises again, forming a U shape whose minimum suggests the right level of capacity or the right number of training epochs. The point where validation loss is minimum motivates early stopping.

What are training, validation, and test splits?

To estimate generalization honestly, the dataset is partitioned into disjoint subsets.

Scroll sideways for more →

Splits should be drawn so that examples are independently and identically distributed (i.i.d.). For time-series problems use a temporal split; for grouped data (patients, users) use a group-aware split to avoid leakage.

What is cross-validation?

When data is scarce, k-fold cross-validation gives a more stable estimate of validation performance. The training set is split into k folds; the model is trained k times, each time holding out one fold as the validation set, and the k validation scores are averaged. Common choices are 5-fold and 10-fold. Stratified k-fold preserves class proportions in each fold and is preferred for class-imbalanced datasets. Leave-one-out cross-validation (LOOCV) is the limiting case where k equals the number of examples; it is unbiased but expensive and high-variance.^[8]

What is a loss function?

A loss function measures the cost of a prediction. Training minimizes the average loss over the training set, sometimes called the empirical risk.^[3]

Scroll sideways for more →

Alongside the per-example loss, training adds a regularization term to the total objective, weighted by a lambda coefficient, which is itself a hyperparameter often tuned by validation.

How do optimization algorithms train a model?

Once the loss is defined, an optimizer iteratively updates the model parameters to reduce it. Almost all modern ML uses first-order gradient methods.

What is gradient descent?

Vanilla gradient descent computes the gradient of the loss over the entire training set and takes a step opposite to that gradient, scaled by a learning rate. For large datasets this is impractical because each step requires a full pass over the data.

What is the difference between stochastic and mini-batch gradient descent?

Stochastic gradient descent (SGD) approximates the full-batch gradient with the gradient computed on a single example. Mini-batch SGD, the workhorse of deep learning, computes the gradient on a small batch of examples (the batch size is a key hyperparameter, typically between 32 and 8,192). Mini-batches give a low-variance estimate of the true gradient while still allowing many updates per epoch.^[5]

What are momentum and adaptive optimizers?

Momentum keeps an exponentially weighted average of past gradients, accelerating descent in consistent directions and damping oscillations. Nesterov accelerated gradient (NAG) computes the gradient at a look-ahead position. AdaGrad adapts the learning rate per parameter using the sum of squared past gradients, suiting sparse features but eventually shrinking the rate to zero. RMSProp replaces that cumulative sum with an exponentially decayed moving average. Adam (Adaptive Moment Estimation), introduced by Kingma and Ba in 2014, combines momentum and RMSProp with bias-corrected moments and is the default for many deep learning tasks.^[13] AdamW decouples weight decay from the gradient step and is preferred for transformers. Learning rate schedules (step decay, cosine decay, warmup followed by decay) are used together with optimizers to improve convergence.

What are the main regularization techniques?

Regularization adds inductive bias toward simpler functions to fight overfitting.

Scroll sideways for more →

The strength of L1 or L2 regularization is controlled by the regularization rate (lambda), tuned on the validation set.

What is feature engineering?

Feature engineering converts raw data into the feature vectors consumed by the model. Even with deep learning, careful preparation of inputs often makes a measurable difference.^[7]

Why scale and normalize features?

Numerical features typically require scaling so that no single feature dominates distance and gradient computations.

Scroll sideways for more →

How are categorical features encoded?

Categorical features must be turned into numbers.

• One-hot encoding creates a 0/1 indicator per category, producing a sparse representation for high-cardinality fields.
• Ordinal encoding assigns integers when categories have a natural order.
• Target encoding replaces a category with a statistic of the target (mean, median) computed within cross-validation folds to avoid leakage.
• Hashing trick maps categories into a fixed-size space, useful at very high cardinalities.
• Learned embeddings, used by an embedding layer in a neural network, map each category to a low-dimensional dense vector.

What are feature selection and feature crossing?

Feature selection removes irrelevant or redundant features to reduce variance and training cost. Methods include filter (mutual information, chi-squared), wrapper (recursive feature elimination), and embedded approaches (L1 regularization, tree feature importance). Feature crosses combine two or more features into a synthetic feature that captures their interaction, for example combining latitude and longitude into a grid cell.

What are linear models?

Linear models are the foundation of supervised learning and remain strong baselines.

• Linear regression predicts a continuous target as a weighted sum of features. The closed-form solution uses the normal equation; large problems use SGD.^[9] With L2 regularization it becomes ridge regression; with L1 it becomes lasso.
• Logistic regression maps a linear combination of features through the sigmoid function to produce a probability for binary classification. Despite its name it is a classifier. The multinomial extension uses softmax and is sometimes called softmax regression. The log-odds (logit) is the inverse of the sigmoid.
• Generalized linear models (GLMs) extend linear regression to other distributions, for example Poisson regression for count data and gamma regression for positive continuous responses.
• Linear discriminant analysis (LDA) fits class-conditional Gaussians with shared covariance and produces linear decision boundaries.

What are tree-based models?

Decision trees split the feature space along axis-aligned thresholds and predict the majority class or mean target in each leaf. They handle mixed feature types, are invariant to monotone transformations, and are easy to inspect.

• Random forests (Breiman, 2001) train many trees on bootstrap samples with feature subsampling and average their predictions. They reduce variance with little tuning.^[10]
• Gradient boosted decision trees (GBDT) fit each new tree to the residual gradient of the loss with respect to the current ensemble. Modern implementations include XGBoost (Chen and Guestrin, 2016)^[11], LightGBM (Microsoft, 2017), and CatBoost (Yandex, 2017). GBDT remains the dominant approach on tabular data.
• Isolation forests are an unsupervised variant used for anomaly detection.

Tree models are non-linear and can capture interactions automatically, which is why they often outperform linear models on tabular tasks.

What are distance-based and kernel models?

• k-nearest neighbors (kNN) is a non-parametric method that classifies an example by majority vote of its k nearest neighbors in feature space, with distance computed under a chosen metric (Euclidean, cosine, Mahalanobis). kNN has no training phase but expensive prediction; structures like KD-trees or approximate nearest neighbor (ANN) indexes accelerate it.
• Support vector machines (SVMs), introduced by Cortes and Vapnik (1995), find the maximum-margin hyperplane separating classes.^[12] The kernel trick allows SVMs to fit non-linear boundaries by implicitly mapping inputs into a higher-dimensional space using a kernel function such as the radial basis function (RBF) or polynomial kernel. SVMs use the hinge loss.
• Kernel ridge regression, Gaussian processes, and kernel PCA generalize the kernel idea beyond classification.

What are probabilistic models?

Probabilistic models represent uncertainty through joint or conditional distributions. Naive Bayes assumes feature independence given the class and applies Bayes' rule; despite the strong assumption it is fast and effective for text classification with bag-of-words features (Multinomial, Bernoulli, Gaussian variants). Bayesian networks are directed acyclic graphs whose nodes are random variables and whose edges encode conditional dependence; they were popularized by Judea Pearl in the 1980s. Hidden Markov models (HMMs) and conditional random fields (CRFs) model sequence data and were workhorses for speech and NLP before neural networks took over. Bayesian linear regression, Gaussian processes, and variational autoencoders (VAEs) bring Bayesian principles to modern ML.^[4]

What is a neural network?

A neural network stacks layers of neurons. Each neuron computes a weighted sum of its inputs plus a bias, then applies a non-linear activation function such as the Rectified Linear Unit (ReLU), sigmoid, or hyperbolic tangent. A network has an input layer, one or more hidden layers, and an output layer; the number of hidden layers gives the model's depth and a network with more than one hidden layer is called a deep model.^[5] Training uses backpropagation, which computes gradients by applying the chain rule from the output back through the layers.^[17] The fundamentals chapter introduces these concepts, while the dedicated neural networks and deep learning chapter covers convolutional^[14], recurrent, and transformer architectures^[15] in more depth.

How are machine learning models evaluated?

The choice of metric depends on the task and on what kinds of errors matter most.

What metrics are used for classification?

Scroll sideways for more →

A confusion matrix is the standard visualization that shows TP, FP, TN, and FN counts and is the source of all the metrics above.

What metrics are used for regression?

Scroll sideways for more →

How are ranking quality and probability calibration measured?

Ranking tasks (search, recommendation) use Mean Reciprocal Rank (MRR), Mean Average Precision (MAP), and Normalized Discounted Cumulative Gain (NDCG). Calibration plots and Expected Calibration Error (ECE) check whether predicted probabilities match observed frequencies.

What are the stages of a machine learning pipeline?

A real ML system is more than a model. The end-to-end pipeline typically includes the following stages.

Scroll sideways for more →

MLOps is the engineering discipline that productionizes this loop, with tools such as MLflow, Kubeflow, Vertex AI, SageMaker, and Tecton handling experiment tracking, feature stores, and continuous training.

How are hyperparameters tuned?

Hyperparameters are configuration values set before training (learning rate, batch size, regularization strength, depth, hidden width, dropout rate). Common search strategies include grid search (all combinations on a discrete grid), random search (often outperforms grid search for the same budget; Bergstra and Bengio, 2012)^[16], Bayesian optimization (a probabilistic surrogate that picks configurations maximizing expected improvement), Hyperband and BOHB (random search with early stopping via successive halving), and population-based training. Tools include Optuna, Ray Tune, Weights and Biases Sweeps, and Vertex AI Vizier.

What is the difference between online and offline learning?

Offline learning trains a model on a fixed snapshot of data and then deploys it; the model is static until the next retraining cycle. Online learning updates the model as new examples arrive, which is essential when the data distribution changes (a property called nonstationarity). Online inference produces predictions on demand, while offline (batch) inference scores entire datasets on a schedule. The choice between static and dynamic inference depends on freshness requirements, latency budgets, and cost.

How is class imbalance handled?

When one class is much more frequent than the other, naive accuracy is misleading. Strategies include resampling (oversampling the minority class with SMOTE or ADASYN, or undersampling the majority class), class weighting (scaling the loss by inverse frequency), tuning the classification threshold, cost-sensitive learning (asymmetric costs in the loss), and anomaly detection framing when the positive class is extremely rare.

What is interpretability and responsible machine learning?

Interpretability matters for debugging, fairness, and regulatory compliance. Common techniques include feature importance from tree ensembles (Gini, permutation, SHAP), partial dependence plots, local surrogates such as LIME (Ribeiro et al., 2016)^[18], SHAP values (Lundberg and Lee, 2017)^[19], counterfactual explanations, and attention visualization for deep models.

Algorithmic bias can creep in through skewed data, label noise, or proxy labels that correlate with protected attributes. Fairness metrics include demographic parity, equalized odds, and predictive parity; no single metric satisfies all desiderata simultaneously (Chouldechova, 2017; Kleinberg et al., 2017).^[20][21] Mitigation spans pre-processing (reweighting), in-processing (constraints), and post-processing (group-specific thresholds). Stability and feedback loops are also material risks. The NIST AI Risk Management Framework (2023)^[23] and EU AI Act (2024)^[24] formalize many of these requirements.

What software and tools are used for machine learning?

Most practitioners use the Python data stack: pandas (and its DataFrame abstraction), NumPy, scikit-learn for classical models, and PyTorch, TensorFlow, or JAX for deep learning. R, Julia, and Spark MLlib are also widely used. ONNX provides a portable format for deployed models, and Hugging Face Hub hosts pretrained checkpoints.

What are the best textbooks and courses for machine learning fundamentals?

The following resources are widely used and define the canon of fundamentals.

Scroll sideways for more →

The Google Machine Learning Glossary, on which much of the terminology in this wiki draws, is also a high-quality starting point.^[22]

Index of fundamentals terms

The terms below have dedicated wiki pages in the fundamentals chapter of Machine learning terms. Where multiple notations exist (for example ReLU and Rectified Linear Unit), separate entries are linked.

• accuracy
• activation function
• artificial intelligence
• AUC (Area Under the Curve)
• backpropagation
• batch
• batch size
• bias
• bias (ethics/fairness)
• binary classification
• bucketing
• categorical data
• class
• classification model
• classification threshold
• class-imbalanced dataset
• clipping
• confusion matrix
• continuous feature
• convergence
• DataFrame
• dataset
• deep model
• dense feature
• depth
• discrete feature
• dynamic
• dynamic model
• early stopping
• embedding layer
• epoch
• example
• false negative (FN)
• false positive (FP)
• false positive rate (FPR)
• feature
• feature cross
• feature engineering
• feature set
• feature vector
• feedback loop
• generalization
• generalization curve
• gradient descent
• ground truth
• hidden layer
• hyperparameter
• independently and identically distributed (i.i.d.)
• inference
• input layer
• interpretability
• iteration
• L0 regularization
• L1 loss
• L1 regularization
• L2 loss
• L2 regularization
• label
• labeled example
• lambda
• layer
• learning rate
• linear model
• linear
• linear regression
• logistic regression
• Log Loss
• log-odds
• loss
• loss curve
• loss function
• machine learning
• majority class
• mini-batch
• minority class
• model
• multi-class classification
• negative class
• neural network
• neuron
• node (neural network)
• nonlinear
• nonstationarity
• normalization
• numerical data
• offline
• offline inference
• one-hot encoding
• one-vs.-all
• online inference
• online learning
• output layer
• overfitting
• pandas
• parameter
• positive class
• post-processing
• prediction
• proxy labels
• rater
• Rectified Linear Unit (ReLU)
• regression model
• regularization
• regularization rate
• ReLU
• ROC (receiver operating characteristic) Curve
• Root Mean Squared Error (RMSE)
• sigmoid function
• softmax
• sparse feature
• sparse representation
• sparse vector
• squared loss
• stability
• static
• static inference
• stationarity
• stochastic gradient descent (SGD)
• supervised machine learning
• synthetic feature
• test loss
• training
• training loss
• training-serving skew
• training set
• true negative (TN)
• true positive (TP)
• true positive rate (TPR)
• underfitting
• unlabeled example
• unsupervised machine learning
• validation
• validation loss
• validation set
• weight
• weighted sum
• Z-score normalization

see also

• Machine learning terms
• Machine learning terms/Neural networks
• Machine learning terms/Language evaluation
• Machine learning terms/Advanced
• Artificial intelligence
• Deep learning
• Neural network

References

1. Mitchell, Tom M. *Machine Learning*. McGraw-Hill, 1997. ↩
2. Hastie, Trevor; Tibshirani, Robert; Friedman, Jerome. *The Elements of Statistical Learning*. 2nd ed. Springer, 2009. ↩
3. Bishop, Christopher M. *Pattern Recognition and Machine Learning*. Springer, 2006. ↩
4. Murphy, Kevin P. *Probabilistic Machine Learning: An Introduction*. MIT Press, 2022. ↩
5. Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron. *Deep Learning*. MIT Press, 2016. ↩
6. Sutton, Richard S.; Barto, Andrew G. *Reinforcement Learning: An Introduction*. 2nd ed. MIT Press, 2018. ↩
7. Geron, Aurelien. *Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow*. 3rd ed. O'Reilly, 2022. ↩
8. James, Gareth; Witten, Daniela; Hastie, Trevor; Tibshirani, Robert. *An Introduction to Statistical Learning*. 2nd ed. Springer, 2021. ↩
9. Deisenroth, Marc Peter; Faisal, A. Aldo; Ong, Cheng Soon. *Mathematics for Machine Learning*. Cambridge University Press, 2020. ↩
10. Breiman, Leo. "Random Forests." *Machine Learning* 45, no. 1 (2001): 5-32. ↩
11. Chen, Tianqi; Guestrin, Carlos. "XGBoost: A Scalable Tree Boosting System." *KDD*, 2016. ↩
12. Cortes, Corinna; Vapnik, Vladimir. "Support-Vector Networks." *Machine Learning* 20, no. 3 (1995): 273-297. ↩
13. Kingma, Diederik P.; Ba, Jimmy. "Adam: A Method for Stochastic Optimization." *ICLR*, 2015. ↩
14. Krizhevsky, Alex; Sutskever, Ilya; Hinton, Geoffrey E. "ImageNet Classification with Deep Convolutional Neural Networks." *NeurIPS*, 2012. ↩
15. Vaswani, Ashish, et al. "Attention Is All You Need." *NeurIPS*, 2017. ↩
16. Bergstra, James; Bengio, Yoshua. "Random Search for Hyper-Parameter Optimization." *Journal of Machine Learning Research* 13 (2012): 281-305. ↩
17. Rumelhart, David E.; Hinton, Geoffrey E.; Williams, Ronald J. "Learning Representations by Back-Propagating Errors." *Nature* 323, no. 6088 (1986): 533-536. ↩
18. Ribeiro, Marco Tulio; Singh, Sameer; Guestrin, Carlos. "Why Should I Trust You? Explaining the Predictions of Any Classifier." *KDD*, 2016. ↩
19. Lundberg, Scott M.; Lee, Su-In. "A Unified Approach to Interpreting Model Predictions." *NeurIPS*, 2017. ↩
20. Chouldechova, Alexandra. "Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments." *Big Data* 5, no. 2 (2017): 153-163. ↩
21. Kleinberg, Jon; Mullainathan, Sendhil; Raghavan, Manish. "Inherent Trade-Offs in the Fair Determination of Risk Scores." *ITCS*, 2017. ↩
22. Google Developers. "Machine Learning Glossary." https://developers.google.com/machine-learning/glossary, accessed 2026. ↩
23. National Institute of Standards and Technology. *AI Risk Management Framework (AI RMF 1.0)*. NIST AI 100-1, 2023. ↩
24. European Union. *Regulation (EU) 2024/1689 (EU AI Act)*. Official Journal of the European Union, 2024. ↩
25. Samuel, Arthur L. "Some Studies in Machine Learning Using the Game of Checkers." *IBM Journal of Research and Development* 3, no. 3 (1959): 210-229. ↩