Model training

Model training is the process by which a machine learning system learns parameter values, model structure, or both from data or interaction. A training procedure connects a model, an objective, an update or fitting rule, and a source of examples or feedback. Its output is a fitted model plus the information needed to evaluate, reproduce, and govern that model. Inference is the later use of the fitted model to produce predictions or other outputs; validation and testing estimate how well it works under specified conditions.^[1][2]

Training is broader than running backpropagation on a neural network. Linear regression may have a closed-form least-squares solution, latent-variable models may use expectation-maximization, and tree learners construct a sequence of splits. Neural networks commonly use minibatch gradient methods, but even there, differentiation and parameter updating are separate operations.^[3][4]

A low training loss is not the final goal by itself. The useful question is whether the trained system performs reliably on data and conditions that represent its intended use. That makes split design, preprocessing, metrics, hyperparameter selection, documentation, and monitoring part of the training process rather than administrative work around it.^[2][5]

For a simple intuition, parameters are adjustable controls inside the model. Training examples expose an error or another feedback signal, and the fitting rule decides how to move those controls. A real training run must also define what the signal means, which examples may influence it, when to stop, and how to tell whether the result works beyond those examples.

What training changes

A parameter is learned during fitting. Examples include a linear model's coefficients, a neural network's weights, a probability distribution's parameters, or a decision tree's split rules. An optimizer state is auxiliary information maintained by an iterative algorithm, such as momentum or an adaptive second-moment estimate. A hyperparameter is chosen outside the parameter-update rule, although an outer optimization process may tune it. Learning rate, regularization strength, tree depth, batch size, and training duration are common hyperparameters.

These boundaries matter operationally. A saved model state may be enough for inference, but PyTorch's general checkpoint pattern for resuming training also saves optimizer state. Its example records the epoch and latest loss, and the documentation explains that optimizer state contains buffers and parameters updated during training.^[25]

Training may begin from a random or analytically chosen state, or from parameters learned in an earlier run. Pre-training fits a reusable model before a later task-specific stage. Fine-tuning continues fitting that model on a new objective or dataset. Houlsby et al. describe fine-tuning a pretrained model as a transfer learning mechanism and propose adapter tuning as a parameter-efficient fine-tuning method: the original network stays fixed while newly introduced adapter modules are trained for a downstream task.^[33] Both full fine-tuning and adapter tuning still require an independent evaluation protocol for the intended task.^[1]

Training settings are often described by the feedback available to the learner. The categories overlap, and a single system may pass through several of them.

Scroll sideways for more →

Other descriptions cut across this table. Online learning updates from a stream; transfer learning starts from parameters learned elsewhere; semi-supervised learning combines labeled and unlabeled examples; active learning chooses which examples to label. These are not mutually exclusive alternatives to supervised or self-supervised training.^[1]

Training workflow

Training is usually an iterative experiment rather than a single command. The order below is useful because it keeps evaluation evidence separate from decisions that could contaminate it.

Define the task, objective, and metrics

The task definition states what the model receives, what it should produce, who or what is affected, and which deployment conditions matter. The training objective converts part of that goal into a quantity the fitting procedure can optimize. A metric is used to judge behavior. They need not be the same.

For example, a classifier may optimize cross-entropy while the deployment decision depends on recall at a fixed false-positive rate. A language model may minimize next-token loss while an application is evaluated for factuality, latency, or unsafe behavior. A surrogate objective is often necessary because the deployment metric is discontinuous, delayed, expensive, or only partially observable.^[4]

An objective also contains policy choices. Example weights, class weights, regularization terms, preference labels, reward design, and filtering rules all change what the model is encouraged to learn. Improving the objective value therefore establishes only that the model improved according to that formulation.

Build the dataset and split by the deployment unit

Training data should be assessed for provenance, collection conditions, licensing or consent constraints, representativeness, labeling procedures, known exclusions, and maintenance. Before any learned preprocessing or feature selection, examples are divided according to the unit that will be independent in deployment.^[6][7][8]

Scroll sideways for more →

A random row split is appropriate only when rows are reasonable independent units for the intended prediction. Time-series systems may need forward-looking splits. Medical data often need patient-level splits. Repeated measurements may need group-level splits. Graph, household, geographic, and near-duplicate records require similar care. Leakage can enter through collection, target construction, aggregation, preprocessing, feature selection, or split design, not only through exact duplicate rows.^[6]

Every learned transformation should be fit on the training partition and then applied to validation and test data. Scaling all rows before splitting, selecting features with the full target vector, or imputing with statistics computed from the test set leaks evaluation information. Pipeline abstractions can make the fit and transform boundary explicit.^[7]

Initialize and fit

An iterative training run begins from an initial parameter state. Initialization can change signal and gradient propagation, especially in deep networks, so the initializer belongs in the experiment record.^[9] Each iteration then performs some version of the following sequence:

1. Obtain a batch or interaction trajectory from the training source.
2. Run the model to compute predictions, latent quantities, or actions.
3. Evaluate the training objective and any diagnostics.
4. Estimate the quantities required by the update rule, such as gradients, sufficient statistics, residuals, or split scores.
5. Apply the parameter update or fitting step.
6. Record loss, metrics, learning rate, throughput, numerical warnings, and resource use.
7. At declared intervals, evaluate on validation data without fitting to it directly.

For a finite dataset, an epoch is one complete pass through the training examples. Epoch counts are less informative for streams, continually sampled environments, or very large corpora, where steps, examples, tokens, interactions, or compute may be the clearer budget.

In a neural-network step, this sequence is often summarized as a forward pass, loss evaluation, backward pass, and optimizer update. The forward pass may retain intermediate activations needed for the backward calculation. Backpropagation computes the required derivatives; it does not itself apply the update.

Stopping may be fixed by a compute budget, convergence criterion, validation rule, or safety limit. Early stopping uses validation behavior to choose a parameter state and therefore functions as a hyperparameter-selection procedure.^[5]

Freeze choices and perform final evaluation

Once the model, preprocessing, hyperparameters, decision threshold, and checkpoint are selected, final evaluation uses the untouched test protocol. Results should include the metric definition, uncertainty or variation across runs where relevant, subgroup or condition-specific results, and known conditions where the estimate does not apply.

The deliverable is more than weights. A usable training artifact normally includes preprocessing logic, feature or tokenizer configuration, model configuration, label mapping, evaluation code, training and data identifiers, and documentation of intended and out-of-scope use. Dataset documentation and model cards provide structured ways to record this information.^[8][10]

Objectives and optimization

For examples indexed by i, a common empirical objective has the form average loss + regularization. The average is calculated over finite training data, while the quantity of interest is often expected loss under an unknown deployment distribution. Minimizing the finite-sample objective can overfit, and a differentiable surrogate may not rank models in exactly the same way as the deployment metric.^[4]

Common objective families

A loss function turns a model output and a target, observation, or preference signal into a quantity used for fitting. The form of that loss changes the tradeoffs the learner makes.

Scroll sideways for more →

The table lists common choices, not fixed pairings. Data noise, class imbalance, calibration needs, robustness goals, and the final decision rule can all justify a different objective. The evaluation metric should therefore be reported separately from the optimized loss.^[3][4]

Minibatches trade a noisy but relatively inexpensive update estimate against the cost of processing more examples at once. Shuffling helps only when the sampling scheme is valid. If nearby records are correlated, blindly shuffling rows can obscure dependence rather than fix it.^[4]

Differentiation, backpropagation, and updates

Backpropagation applies the chain rule efficiently through a composed network to obtain derivatives of an objective with respect to intermediate values and parameters.^[11] Reverse-mode automatic differentiation generalizes this idea to programs composed of differentiable operations and is well suited to scalar objectives with many parameters.^[12]

Neither backpropagation nor automatic differentiation decides the update. Stochastic gradient descent, momentum, Adam, or another optimizer consumes gradients and produces parameter changes. Keeping these steps distinct makes it easier to diagnose failures: a gradient may be wrong, numerically unstable, poorly scaled, or correct but paired with an unsuitable update schedule.

Adam maintains bias-corrected exponential estimates of the first moment and second raw moment of stochastic gradients and uses them to scale parameter-wise updates.^[13] Its original convergence analysis was conditional. Later work constructed simple convex settings in which Adam does not converge and proposed AMSGrad-style changes.^[14] This does not make Adam unusable; it means optimizer choice and tuning remain empirical, and universal convergence claims are not warranted.

Weight decay also needs precise language. For ordinary stochastic gradient descent, an L2 penalty can be made equivalent to a scaled weight-decay update. That equivalence does not generally hold for adaptive optimizers. AdamW decouples weight decay from the adaptive loss-gradient step.^[15]

Training without gradient descent

Not every model needs backpropagation or even an iterative gradient optimizer. Ordinary least squares may be solved through linear algebra. Some probabilistic models use maximum-likelihood or Bayesian inference. Expectation-maximization alternates between expectations over latent quantities and parameter maximization. Decision trees search for splits, while boosting fits a sequence of weak learners to residual or gradient-like signals. The unifying idea is fitted model state, not one numerical procedure.^[3]

Generalization and regularization

Generalization is performance on relevant unseen cases. Training error and generalization error can move differently, so a training curve must be read together with validation evidence.^[2]

Scroll sideways for more →

Regularization modifies the learning process to reduce generalization error rather than merely lower training error.^[5] Common methods include parameter penalties, data augmentation, early stopping, capacity constraints, and stochastic techniques such as dropout. Dropout randomly omits units during training and was introduced as a way to reduce overfitting in neural networks, but its usefulness and rate are architecture- and task-dependent.^[16]

Initialization and normalization can also improve optimization without being regularizers in the same sense. The original Batch Normalization paper attributed much of its benefit to reduced internal covariate shift.^[17] Later controlled experiments found little connection between that measured distributional stability and the optimization benefit, and instead argued that BatchNorm smooths the optimization landscape.^[18] The causal explanation remains something to attribute and test, not a settled one-line fact.

Regularization should be selected with the validation protocol and deployment constraints in view. A smaller gap on one validation set does not establish robustness to a different population, time period, sensor, language, or attack.

Hyperparameter selection

Hyperparameter tuning includes the search space, sampling strategy, budget, stopping rule, and selection metric. Grid search spends equal effort on every listed dimension. Random search can be more efficient when only a subset of dimensions strongly affects validation performance.^[19] Bayesian optimization, population-based methods, and learned schedulers add assumptions and overhead, so their benefit depends on evaluation cost and search geometry.

Fair comparisons hold the tuning protocol and resource budget as constant as possible. Reporting only the best run hides the number of attempts and can favor methods given broader search spaces. At a minimum, record the full search space, number of trials, seed policy, early-termination policy, selection rule, and compute used.

Repeated model selection consumes information from the validation set. If a benchmark or test set influences architecture, prompts, preprocessing, thresholds, or data collection, it no longer provides an untouched final estimate. A new holdout, nested cross-validation design, or independent evaluation may be needed.

Scaling and training systems

Large distributed training runs divide computation, memory, or both. These changes can alter the optimization path as well as wall-clock time.

Scroll sideways for more →

DistBelief demonstrated both model-parallel execution and replicated, asynchronous data-parallel optimization at cluster scale.^[20] GPipe later described synchronous pipeline parallelism in which minibatches are divided into microbatches and gradients are accumulated before an update.^[21] ZeRO partitions optimizer state, gradients, and parameters to reduce data-parallel memory redundancy.^[22] These systems illustrate different points in the design space, not interchangeable labels.

Increasing data parallelism commonly increases the global batch size. That can reduce the number of serial update steps for a while, but the relationship is workload-dependent. A large study across 35 neural-network workloads found very large variation in the batch-size versus step-count tradeoff and showed that tuning and compute budgets explained some apparent disagreements.^[23] A large batch is therefore a capacity and optimization choice, not only a throughput setting.

Mixed-precision training reduces memory traffic and can exploit faster low-precision hardware. The original widely used FP16 recipe retained FP32 master weights, scaled the loss to preserve small gradients, and used FP32 for selected reductions.^[24] Speedups and numerical behavior depend on hardware, operators, model, and dynamic range. Overflow, underflow, and skipped updates should be monitored rather than assumed away.

Durable checkpoints support recovery and resumption. PyTorch's general checkpoint pattern stores model and optimizer state and records the epoch and latest loss.^[25] The optimizer state matters because it contains buffers and parameters updated during training. A model-only file can restore weights for inference without restoring the optimizer state used to continue training.^[25]

Empirical scaling laws can guide resource allocation within the regimes that produced them. Kaplan and colleagues fit power-law relationships for autoregressive language-model loss as a function of model size, data, and compute.^[26] The later Chinchilla scaling experiments found a different compute-optimal allocation, with parameters and training tokens increasing together across the studied range.^[27] These are scoped empirical results for particular model families and objectives, not universal ratios for all machine learning.

Reproducibility and run records

Reproducibility has several levels. A rerun may aim for bitwise-identical outputs, statistically similar results in the same software stack, or an independent replication of the scientific conclusion. Those goals require different evidence.

Setting seeds is necessary for many controlled comparisons, but it is not sufficient. Framework documentation warns that identical results are not guaranteed across releases, platforms, or CPU and GPU execution even with the same seeds. Deterministic algorithms can also be slower.^[28] Floating-point operation order, nondeterministic kernels, data-loader scheduling, distributed collectives, and hardware libraries can all change a trajectory.

A useful run record includes:

• immutable identifiers or hashes for raw data, processed data, code, and configuration;
• model architecture, initialization, objective, optimizer, schedules, regularization, batch construction, and stopping rule;
• random seeds and the libraries or components to which each seed applies;
• software packages, drivers, compiler settings, hardware type, device count, and distributed topology;
• training and validation curves, selected checkpoint, failed runs, and numerical warnings;
• search space, trial budget, selection metric, and per-run resource use;
• final evaluation protocol, uncertainty across runs, and known deviations from the declared plan.

Community reproducibility initiatives have promoted code policies, checklists, and independent reruns, but evidence that any single intervention improves research quality is limited.^[29] The run record should make inspection and rerunning possible without pretending that documentation guarantees replication.

Security, privacy, and governance

Training data and infrastructure create attack surfaces. Data poisoning attacks attempt to alter training behavior through manipulated examples, labels, feedback, or update paths. Supply-chain compromise can affect pretrained weights, data-processing code, dependencies, or distributed workers. Leakage can expose target information during evaluation, while privacy attacks can reveal information about the training data after deployment.^[1]

Membership inference asks whether a particular record was in a model's training set. Black-box attacks have demonstrated that this information can sometimes be inferred from model outputs.^[30] Risk varies with the model, data, access, overfitting, and attack design, so one experiment does not imply equal leakage from every trained model.

Differentially private stochastic gradient descent clips per-example gradients, adds calibrated noise, and accounts for privacy loss over repeated updates.^[31] Its guarantee depends on the adjacency definition, sampling, clipping threshold, noise, accounting method, and reported privacy parameters. Privacy usually trades off with utility, compute, or both, and a training method does not address every downstream privacy risk.

Governance begins before fitting. Dataset documentation can record motivation, composition, collection, processing, recommended uses, distribution, and maintenance.^[8] Model cards can record intended and out-of-scope uses, evaluation data and metrics, disaggregated results, ethical considerations, and limitations.^[10] These artifacts improve traceability, but they do not prove consent, fairness, safety, or legal compliance.

The NIST AI Risk Management Framework calls for documenting data availability, representativeness, and suitability; defining test, evaluation, verification, and validation methods; evaluating under conditions similar to deployment; documenting limitations; and monitoring production behavior.^[32] It is voluntary risk-management guidance, not a certification. The practical principle is broader: training decisions and evaluation evidence should remain reviewable throughout the system lifecycle.

Practical checklist

Scroll sideways for more →

See also

• Machine learning terms
• Learning rate
• Pre-training
• Fine Tuning
• Inference

References

1. ^NIST, *Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations*, NIST AI 100-2e2025, 2025. doi.org/...NIST.AI.100-2e2025
2. ^Ian Goodfellow, Yoshua Bengio, and Aaron Courville, *Deep Learning*, Chapter 5, "Machine Learning Basics," 2016. deeplearningbook.org/...ml
3. ^Christopher M. Bishop, *Pattern Recognition and Machine Learning*, 2006. microsoft.com/...tern-recognition-machine-learning
4. ^Ian Goodfellow, Yoshua Bengio, and Aaron Courville, *Deep Learning*, Chapter 8, "Optimization for Training Deep Models," 2016. deeplearningbook.org/...optimization
5. ^Ian Goodfellow, Yoshua Bengio, and Aaron Courville, *Deep Learning*, Chapter 7, "Regularization for Deep Learning," 2016. deeplearningbook.org/...regularization
6. ^Shachar Kaufman et al., "Leakage in Data Mining: Formulation, Detection, and Avoidance," *ACM Transactions on Knowledge Discovery from Data*, 2012. doi.org/...2382577.2382579
7. ^scikit-learn, "Common pitfalls and recommended practices," version 1.7. scikit-learn.org/...common_pitfalls
8. ^Timnit Gebru et al., "Datasheets for Datasets," *Communications of the ACM*, 2021. doi.org/...3458723
9. ^Xavier Glorot and Yoshua Bengio, "Understanding the difficulty of training deep feedforward neural networks," AISTATS 2010. proceedings.mlr.press/...glorot10a
10. ^Margaret Mitchell et al., "Model Cards for Model Reporting," FAT* 2019. doi.org/...3287560.3287596
11. ^David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams, "Learning representations by back-propagating errors," *Nature*, 1986. doi.org/...323533a0
12. ^Atilim Gunes Baydin et al., "Automatic Differentiation in Machine Learning: a Survey," *Journal of Machine Learning Research*, 2018. jmlr.org/...17-468
13. ^Diederik P. Kingma and Jimmy Ba, "Adam: A Method for Stochastic Optimization," ICLR 2015. arxiv.org/...1412.6980
14. ^Sashank J. Reddi, Satyen Kale, and Sanjiv Kumar, "On the Convergence of Adam and Beyond," ICLR 2018. arxiv.org/...1904.09237
15. ^Ilya Loshchilov and Frank Hutter, "Decoupled Weight Decay Regularization," ICLR 2019. arxiv.org/...1711.05101
16. ^Nitish Srivastava et al., "Dropout: A Simple Way to Prevent Neural Networks from Overfitting," *Journal of Machine Learning Research*, 2014. jmlr.org/...srivastava14a
17. ^Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift," ICML 2015. proceedings.mlr.press/...ioffe15
18. ^Shibani Santurkar et al., "How Does Batch Normalization Help Optimization?" NeurIPS 2018. proceedings.neurips.cc/...560467e0a99e1cf-Abstract
19. ^James Bergstra and Yoshua Bengio, "Random Search for Hyper-Parameter Optimization," *Journal of Machine Learning Research*, 2012. jmlr.org/...bergstra12a
20. ^Jeffrey Dean et al., "Large Scale Distributed Deep Networks," NeurIPS 2012. proceedings.neurips.cc/...823815f66102863-Abstract
21. ^Yanping Huang et al., "GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism," NeurIPS 2019. proceedings.neurips.cc/...76b1c5722a46aa2-Abstract
22. ^Samyam Rajbhandari et al., "ZeRO: Memory Optimizations Toward Training Trillion Parameter Models," SC20. doi.org/...SC41405.2020.00024
23. ^Christopher J. Shallue et al., "Measuring the Effects of Data Parallelism on Neural Network Training," *Journal of Machine Learning Research*, 2019. arxiv.org/...1811.03600
24. ^Paulius Micikevicius et al., "Mixed Precision Training," ICLR 2018. arxiv.org/...1710.03740
25. ^PyTorch, "Saving and Loading Models." docs.pytorch.org/...saving_loading_models
26. ^Jared Kaplan et al., "Scaling Laws for Neural Language Models," 2020. arxiv.org/...2001.08361
27. ^Jordan Hoffmann et al., "Training Compute-Optimal Large Language Models," 2022. arxiv.org/...2203.15556
28. ^PyTorch, "Reproducibility," PyTorch 2.12 documentation, updated 2025-10-03. docs.pytorch.org/...randomness
29. ^Joelle Pineau et al., "Improving Reproducibility in Machine Learning Research: a Report from the NeurIPS 2019 Reproducibility Program," *Journal of Machine Learning Research*, 2021. jmlr.org/...20-303
30. ^Reza Shokri et al., "Membership Inference Attacks Against Machine Learning Models," IEEE Symposium on Security and Privacy, 2017. doi.org/...SP.2017.41
31. ^Martin Abadi et al., "Deep Learning with Differential Privacy," ACM CCS 2016. arxiv.org/...1607.00133
32. ^NIST, *Artificial Intelligence Risk Management Framework (AI RMF 1.0)*, 2023. doi.org/...NIST.AI.100-1
33. ^Neil Houlsby et al., "Parameter-Efficient Transfer Learning for NLP," ICML 2019. proceedings.mlr.press/...houlsby19a