Overfitting

Overfitting is a failure of generalization: a fitted model or a model-selection procedure performs better on the observations used to develop it than on new observations from the population of interest. The central issue is not merely that a model has many parameters, fits its training set exactly, or attains low training loss. Overfitting is about the difference between performance on development data and the performance required after development, under a stated sampling and deployment setting.^[1][2]

A training-to-evaluation gap can be evidence of overfitting, but it is not a self-interpreting diagnosis. The observed gap is a finite-sample estimate. It can be distorted by a small validation sample, repeated tuning on the same holdout, duplicate or related records across splits, a mismatch between metrics, or distribution shift. Conversely, a small gap can occur because both training and evaluation performance are poor. Sound diagnosis therefore requires a defined target population, independent evaluation data, and a record of every data-dependent choice.^{[1][6][9][10]}

Overfitting is often associated with high variance in classical statistical examples, but the two terms are not interchangeable in every setting. Likewise, exact interpolation of the training data is not by itself proof of harmful overfitting. Research on modern overparameterized models documents cases of double descent, benign interpolation, and delayed generalization, all under particular data, model, and optimization conditions.^{[23][24][25][26][27]} These results refine the classical account; they do not make independent evaluation or regularization unnecessary.

Formal definition

Let a learning algorithm use a sample $D=((x_1,y_1),\ldots,(x_n,y_n))$ to produce a predictor $f_D$. For a nonnegative loss function $\ell$ and a target distribution $P$ over examples, the population risk is

$$R_P(f_D)=\mathbb{E}_{(X,Y)\sim P}\left(\ell(f_D(X),Y)\right).$$

The empirical risk on the training sample is

$$\widehat{R}_D(f_D)=\frac{1}{n}\sum_{i=1}^{n}\ell(f_D(x_i),y_i).$$

The population generalization gap for that fitted predictor is

$$G_D=R_P(f_D)-\widehat{R}_D(f_D).$$

In practice, $R_P$ is unknown. A held-out validation set or test set supplies another estimate, not the population risk itself. If $T$ is an independent sample from the same target distribution, then

$$\widehat{G}_{D,T}=\widehat{R}_T(f_D)-\widehat{R}_D(f_D)$$

is an empirical gap. Its magnitude depends on sample sizes, the loss scale, class balance, dependence among records, and sampling variation. There is no universal accuracy difference or loss difference above which every model is overfit.^[1][2]

This formulation also exposes three distinct objects:

• Fitting overfitting occurs when the learning procedure adapts too specifically to its training observations.
• Selection overfitting occurs when alternatives, features, thresholds, or hyperparameters are repeatedly compared on a validation criterion until the selected result exploits noise in that criterion.^[6]
• Evaluation failure occurs when the held-out estimate is not valid for the intended claim, for example because preprocessing crossed the split boundary or the deployment distribution differs from the evaluation distribution.^[9][10]

Only the first two are overfitting in the narrow sense. The third can produce the same pattern of disappointing deployment performance and must be ruled out before assigning a cause.

Relation to empirical risk minimization

Empirical risk minimization chooses a predictor from a hypothesis class to minimize training risk. The empirical objective is useful because the population risk is not directly available, but an unrestricted or poorly controlled search can fit accidental properties of a finite sample. Learning theory studies conditions under which empirical performance controls population performance, including restrictions on the hypothesis class, regularization, margins, stability, and sample complexity.^[2][5]

Capacity is therefore relative to more than a parameter count. The relevant model capacity depends on the hypothesis class, parameterization, optimization algorithm, regularization, data representation, loss, and distribution. Two procedures with the same nominal number of parameters can have different effective behavior. Conversely, a model with more parameters than observations can sometimes generalize well because the training algorithm favors a particular interpolating solution.^[23][24][26]

Bias and variance

The bias-variance tradeoff gives a useful account of some overfitting phenomena, but its familiar three-term equation is scoped to squared-error prediction under stated assumptions. Suppose

$$Y=f^*(X)+\varepsilon,$$

where $\mathbb{E}(\varepsilon\mid X=x)=0$ and $\operatorname{Var}(\varepsilon\mid X=x)=\sigma^2$. For training samples $D$ drawn by the same sampling process, the expected prediction error at a fixed input $x$ can be written as

$$\mathbb{E}_{D,\varepsilon}\left((Y-f_D(x))^2\right) = \left(\mathbb{E}_D(f_D(x))-f^*(x)\right)^2 + \mathbb{E}_D\left((f_D(x)-\mathbb{E}_D(f_D(x)))^2\right) + \sigma^2.$$

The terms are squared bias, variance across training samples, and irreducible noise. In many classical model families, increasing flexibility reduces bias but raises variance, producing a U-shaped test-risk curve. High variance can then be a mechanism of overfitting.^[3][4]

The decomposition does not say that every overfit model has low bias, that every high-variance procedure is overfit on a particular dataset, or that risk must increase monotonically once a parameter count passes a fixed threshold. Classification losses, dependent observations, misspecification, distribution shift, adaptive selection, and overparameterized interpolation require additional analysis. VC dimension, algorithmic stability, norms, margins, compression, and data-dependent complexity are among the tools used to study generalization beyond the simple diagram.^[2][5][23]

What counts as evidence

Training and evaluation curves

For an iterative learner, plotting training and validation metrics against update count or epoch can reveal divergence. A common pattern is falling training loss while validation loss stops improving or rises. This pattern supports a local statement: later checkpoints perform worse on that validation sample under that metric. It does not establish that every later checkpoint has higher population risk, and it does not identify why the validation estimate changed.

Several checks make the interpretation stronger:

• Training and evaluation must use comparable losses and preprocessing. A model evaluated with augmentation, dropout, or a different normalization mode can create an artificial gap.
• The validation observations must be independent at the relevant unit. Images from the same patient, frames from one video, rows from one household, or future observations from the same time series may not be independent.
• The validation set must not have become a hidden training set through repeated experiments. Recording how many configurations were tried is part of interpreting the winning result.
• Uncertainty should be reported. A difference smaller than the variation across seeds, folds, groups, or sampling intervals is weak evidence.
• Absolute performance matters. Low training and low validation performance with a small gap is usually underfitting, an optimization problem, a data problem, or a metric problem rather than successful generalization.

No universal rule such as "99 percent training accuracy means overfitting" is valid. A deterministic, separable task can legitimately yield perfect training and test accuracy. A difficult or noisy task can overfit while training accuracy remains far below 99 percent. The comparison must be made against a justified out-of-sample estimate.

Learning curves by sample size

A second learning curve varies the number of training observations while holding the evaluation protocol fixed. A large train-evaluation gap that narrows as representative data are added is consistent with a variance or sample-size limitation. If both curves remain poor and close, more capacity, better features, or a better-specified task may be needed. If evaluation performance stops improving, the limitation may instead be label noise, irreducible ambiguity, shift, or a mismatch between the metric and the intended use.

Learning curves must be constructed without leaking information. Feature selection, normalization, imputation, augmentation policy choice, and hyperparameter tuning belong inside each training resample. Performing those operations once on the complete dataset can let the nominally held-out observations influence the fitted pipeline.^[7][9]

Stability and sensitivity

An unstable procedure changes substantially when one observation or a small subset changes. Bousquet and Elisseeff formalized several notions of algorithmic stability and related them to generalization bounds for regularization-based algorithms.^[5] Hardt, Recht, and Singer later analyzed stability of stochastic gradient methods under Lipschitz and smoothness assumptions, deriving results for convex and nonconvex objectives.^[16] These are conditional theoretical results, not a claim that every stable model generalizes or every unstable model fails.

Practical sensitivity checks can include:

• refitting across multiple random seeds and reporting the distribution of evaluation results;
• repeating grouped or time-aware splits when the scientific question permits it;
• measuring how predictions or selected features change under plausible resampling;
• testing whether conclusions depend on a few influential examples;
• comparing the selected configuration with simpler, prespecified baselines.

Instability is particularly important when a reported improvement is small relative to variation across fits.

The evaluation target

"Unseen data" is incomplete unless the source of those data is specified. A random split estimates performance for new independent observations from approximately the same data-generating process. It does not automatically estimate performance at a new hospital, future time period, geographic region, device, user population, or policy environment.

Distribution shift refers to a difference between training and target distributions. Moreno-Torres and colleagues review forms of dataset shift in classification, including changes in input, label, or class-conditional distributions.^[10] Poor performance under shift is not, by itself, proof that a model overfit its training sample. A model can generalize well in distribution and still fail after the population changes. Evaluation should match the deployment unit and time horizon, and results should distinguish in-distribution overfitting from out-of-distribution robustness.

How evaluation itself becomes overfit

Validation reuse

A validation set is part of development once its results influence a choice. Selecting architectures, features, prompts, thresholds, checkpoints, augmentation policies, or random seeds on that set consumes information about it. With enough adaptive comparisons, a team can find a configuration that fits idiosyncrasies of the validation sample even though no gradient was computed on those examples.^[6][8]

Cawley and Talbot showed that variance in a model-selection criterion can lead to overfitting of the criterion and selection bias in performance evaluation.^[6] Dwork and colleagues studied adaptive data analysis and proposed a reusable holdout mechanism that limits information released from the holdout.^[8] The practical lesson is straightforward: do not describe the best validation result as an unbiased test result.

A robust workflow separates roles:

1. The training subset estimates model parameters.
2. The validation subset or inner cross-validation loop makes data-dependent development choices.
3. A locked test subset or outer loop estimates the performance of the complete selection procedure.
4. External or prospective data address transport to a new setting when that is part of the claim.

The final model may be retrained on more data after evaluation, but then the test result estimates the prescribed training procedure, not the exact retrained parameter vector.

Nested cross-validation

When data are too limited for a single test split, nested cross-validation separates selection from assessment. Each outer training fold contains a complete inner tuning process. The outer held-out fold is used only after the configuration is selected. Varma and Simon demonstrated in classification simulations that using the same cross-validation result both to tune a classifier and estimate its error produced optimistic bias, while nested cross-validation substantially reduced that bias in their studied settings.^[7]

Nested cross-validation is not magic. Groups, time order, preprocessing, feature selection, and class imbalance must be handled within the correct loop. The report should state the estimand, split construction, number of candidates, selection rule, and uncertainty method.

Leakage

Data leakage occurs when information unavailable at the intended prediction time influences training or evaluation. Examples include computing normalization statistics on all observations, selecting features before splitting, allowing the same entity into training and test data, using future records to predict the past, or including a feature that directly encodes the outcome.

Leakage can make training and evaluation scores both look strong, so it need not create the familiar large gap. Kapoor and Narayanan documented leakage as a recurring source of irreproducible claims across machine-learning-based sciences and proposed reporting practices designed to expose it.^[9] A leakage audit should precede any conclusion that a model has, or has not, overfit.

Factors that increase overfitting risk

Overfitting is an interaction among data, hypothesis class, objective, optimization, and selection. None of the following factors is sufficient on its own.

Flexible search relative to information

A flexible hypothesis class supplies many ways to match a finite sample. Risk rises when the effective search is large relative to the amount and diversity of information available. This can arise from many parameters, high-degree basis functions, deep trees, extensive feature construction, a large hyperparameter search, or repeated manual experimentation. The relevant quantity is not a universal examples-to-parameters ratio. Modern networks often have more parameters than training examples, yet parameter sharing, optimization bias, norms, augmentation, and data structure constrain which solutions are reached.^[23][24]

The number of degrees of freedom can be more informative than a raw parameter count for some estimators, but even effective degrees of freedom is model- and procedure-specific. Claims about adequate sample size should be justified for the task, noise level, dependence structure, target error, and learning algorithm rather than borrowed from a fixed rule of thumb.

Small, narrow, or dependent samples

Few independent sampling units make empirical patterns less reliable. A dataset can contain millions of rows but little independent information if rows are repeated measurements of the same subjects or near-duplicates. Narrow coverage also lets a model rely on incidental context, such as a scanner, background, author, or collection period that correlates with the label only in development data.

Adding representative observations often reduces uncertainty, but more rows do not repair biased sampling, mislabeled targets, duplicated entities, or a missing deployment subgroup. Data collection should increase coverage of the cases on which the model is expected to operate.

Noise and ambiguity

Flexible models can fit mislabeled observations, measurement error, and accidental correlations. Zhang and colleagues showed experimentally that standard convolutional networks used in their study could reach zero training error after labels were randomized and could also fit random inputs.^[23] The result demonstrates substantial effective capacity for those architectures and training procedures. It does not show that every neural network can memorize every arbitrary dataset, nor that fitting clean structured data is merely memorization.

Noise is not always removable. Some labels are inherently ambiguous or reflect disagreement among annotators. If the target itself is unstable, a model can appear to overfit one labeling process while matching another. Reports should describe label provenance, adjudication, missingness, and estimated disagreement instead of treating every discrepancy as model error.

Irrelevant or weakly justified features

High-dimensional representations give a search procedure more opportunities to find sample-specific correlations. This is related to, but not identical with, the curse of dimensionality. Feature selection can reduce variance, but selecting features on all data before evaluation leaks outcome information. Domain knowledge, prespecification, penalization, and selection performed within resampling are safer than choosing features after examining test results.

Prolonged or adaptive optimization

For some iterative learners, later updates continue reducing training error after validation performance stops improving. Early stopping can then act as a capacity control. It is wrong, however, to claim that every additional epoch after a validation peak necessarily worsens population performance. Validation curves fluctuate, and epoch-wise double descent or grokking can produce later improvements in particular experiments.^[25][27]

The checkpoint is itself a hyperparameter. If it is selected from the validation trajectory, its reported performance must be assessed on data not used to choose it.

Manifestations across model families

The same generalization failure can arise through different mechanisms.

Scroll sideways for more →

These are tendencies, not diagnoses. A small-neighborhood rule can be suitable for some tasks, a deep tree can represent a deterministic rule, and a very large neural network can generalize. The evaluation protocol decides whether a fitted instance succeeds on the stated target.

Mitigation strategies

There is no universally best regularization method.^[1] A mitigation should address the observed failure mode and be selected without contaminating the final assessment.

Improve the data and split

The most direct interventions often occur before changing the model:

• collect more independent, representative sampling units;
• correct label errors and document unavoidable ambiguity;
• deduplicate before splitting, using entity identifiers or similarity checks;
• split by subject, site, time, geography, or another deployment-relevant unit;
• fit preprocessing only on the training portion of each split;
• reserve a genuinely untouched test set or use an outer resampling loop;
• include prespecified subgroup and shift evaluations;
• use simple baselines to determine whether added complexity produces a reproducible gain.

More data are useful only when they add information relevant to the target population. Repeating the same narrow observations can reinforce a shortcut rather than remove it.

Penalized objectives

Penalization changes the objective from empirical fit alone to empirical fit plus a complexity-dependent term. For parameters $w$, an L1-penalized objective has the form

$$\min_w \frac{1}{n}\sum_{i=1}^{n}\ell(f_w(x_i),y_i)+\lambda\lVert w\rVert_1.$$

For least-squares linear regression, this is the lasso introduced by Tibshirani.^[11] L1 penalties can produce zero coefficients, but whether that gives reliable feature selection depends on the design, noise, penalty, and sampling process.

An L2-penalized objective has the form

$$\min_w \frac{1}{n}\sum_{i=1}^{n}\ell(f_w(x_i),y_i)+\lambda\lVert w\rVert_2^2.$$

In linear least squares this is ridge regression. Elastic net combines L1 and L2 terms and was developed to address prediction and variable-selection behavior, including settings with correlated predictors.^[12] The penalty strength must be chosen within the development procedure.

Weight decay multiplies parameters by a shrinkage factor during optimization. For ordinary stochastic gradient descent with a fixed scalar learning rate, a common implementation is closely related to an L2 penalty. The equivalence does not generally hold for adaptive gradient methods because coordinate-wise rescaling also changes the effect of the L2 gradient. AdamW was proposed to decouple weight decay from the loss-gradient update in adaptive optimization.^[20] Descriptions should state the optimizer and implementation rather than treating L2 regularization and weight decay as universal synonyms.

Dropout

The original dropout method randomly omits units and their connections during neural-network training. Srivastava and colleagues motivated it as a way to reduce co-adaptation and described prediction as an approximation to averaging many thinned networks.^[13] Dropout is one option within dropout regularization; its benefit and best rate depend on architecture, dataset, normalization, and training recipe.

Inference scaling is implementation-dependent. In the original presentation, activations or outgoing weights are adjusted to account for retention probability. Many modern libraries use inverted dropout, scaling retained activations during training so that no extra scaling is needed at inference. A technical description should not claim that all implementations activate every unit and scale outputs in the same way.

Early stopping

Early stopping selects a point on an optimization trajectory using development performance. Prechelt documented practical stopping criteria for neural-network training.^[14] In certain inverse problems and reproducing-kernel settings, stopping time can be analyzed as a regularization parameter rather than merely a computational convenience.^[15]

A defensible procedure specifies the monitored metric, evaluation frequency, patience or selection rule, minimum improvement, and checkpoint restoration behavior before inspecting the final test set. The validation set used to select a checkpoint is development data. Early stopping often helps, but it can be premature in regimes that exhibit nonmonotonic test error or delayed generalization. This exception is a reason to validate the stopping rule, not a reason to train indefinitely.

Data augmentation

Data augmentation expands the training distribution with transformations intended to preserve task-relevant meaning. Examples include geometric transformations in computer vision, carefully defined perturbations in audio, or transformations based on known physical symmetries. The transformed sample is valid only if its label or target changes in the prescribed way.

Bishop showed that, under a sum-of-squares objective and small input noise, training with noise has a regularization effect related to a Tikhonov term.^[17] Mixup trains on convex combinations of example pairs and their labels and reported improved generalization in the experiments of Zhang and colleagues.^[21] More recent theory studies augmentation as a way to exploit known invariances and analyzes when subsets of a symmetry group can retain statistical benefits in particular estimator classes.^[22]

Augmentation can also be wrong. Horizontal flips change labels for some writing or medical laterality tasks; time stretching can change a target defined by tempo; synonym replacement can change meaning. Augmentations should be justified by the task, applied only inside training, and evaluated for subgroup effects and distribution realism.

Ensembles

Ensemble methods combine predictors. Breiman's bagging constructs bootstrap replicates and averages or votes across fitted predictors; it was designed to improve unstable procedures by reducing variance.^[18] Random forests add randomized feature selection to an ensemble of trees, with their behavior depending on tree strength and correlation.^[19]

Ensembling does not guarantee protection from overfitting. Correlated models can repeat the same shortcut, boosting can fit noise, and choosing ensemble members on a reused holdout can overfit selection. A valid comparison includes the complete ensemble-building procedure within the evaluation protocol.

Capacity and architecture

Reducing depth, width, polynomial degree, tree depth, or the feature set can help when the current procedure is too flexible for the available information. It is not a universal rule that a smaller model generalizes better. A smaller model can underfit, and a larger parameterization can sometimes make optimization favor smoother or lower-norm interpolants.^[24][26]

Pruning should be treated as an algorithm with its own selection criterion. Tree pruning has a direct structural role, while neural-network pruning is also used for compression and efficiency. Removing parameters after training does not automatically improve population risk, and sparse subnetworks found in one architecture or dataset do not establish a general percentage by which all networks should be reduced.

Batch normalization can change optimization and may affect generalization in some experiments, including the reinforcement-learning study discussed below.^[30] It should not be listed as a generic anti-overfitting guarantee. Its effects depend on batch statistics, architecture, batch size, and train-to-inference behavior.

Optimization and implicit regularization

The optimization algorithm determines which solution is selected when many parameters fit the data. Analyses of stochastic gradient descent cover particular objectives and assumptions, including stability bounds and minimum-norm behavior in some linear problems.^[16][26] These results do not support the blanket claim that SGD always converges to the minimum-norm solution in every overparameterized neural network.

When implicit bias matters, the training recipe is part of the model definition. Learning-rate schedule, initialization, batch construction, optimizer, stopping rule, explicit penalties, precision, and data order can all affect the selected predictor. Reproducible evaluation records them rather than attributing generalization to parameter count alone.

Modern overparameterized learning

Memorization does not fully explain generalization

The experiments of Zhang and colleagues showed that neural networks capable of fitting real labels could also fit randomized labels and randomized inputs in the studied image-classification settings.^[23] They also found that commonly used explicit regularizers were not sufficient to explain the observed generalization behavior. The paper challenged explanations based only on nominal capacity.

Two conclusions must remain separate:

• Zero training error demonstrates interpolation, not automatically useful generalization.
• The ability to memorize a corrupted dataset does not imply that the predictor trained on structured data uses the same representation or decision rule.

Architecture, data structure, optimizer, margins, norms, stability, and augmentation can all affect which interpolating solution is selected.

Double descent

Double descent describes a nonmonotonic risk curve in which test risk first follows a classical U-shaped pattern, peaks near an interpolation threshold, and then falls again as effective capacity increases. Belkin and colleagues presented this unified curve and experiments with several model families and datasets.^[24] Their interpolation threshold is the point at which the fitted predictor first achieves approximately zero training risk, not a universal equality between raw parameter count and sample count.

Nakkiran and colleagues, including researchers affiliated with OpenAI, reported model-wise, epoch-wise, and sample-wise double-descent behavior in a collection of deep-learning experiments.^[25] One prominent figure used ResNet-18 models on CIFAR-10 with 15 percent label noise, data augmentation, Adam, and up to 4,000 epochs. In that setting, test error varied nonmonotonically with both width and training time. Such evidence establishes that monotonic textbook heuristics can fail. It does not show that every model, dataset, or deep learning training run has a second descent.

Practical consequences are limited but important:

• Evaluate complexity on both sides of any observed interpolation threshold rather than assuming the first validation minimum is globally optimal.
• Do not infer effective capacity from parameter count alone.
• Do not deliberately cross an interpolation peak without validation evidence for the same task.
• Preserve an untouched assessment set, because choosing a regime from the validation curve is itself model selection.

Claims that a large language model must be safely "far beyond" its interpolation threshold are usually not justified by parameter and token counts alone. Repeated tokens, sequence dependence, objectives, scaling, and the definition of an independently constrained training equation complicate that comparison.^[24][25]

Benign overfitting

Benign overfitting is the coexistence of exact fit to noisy training data and near-optimal prediction risk in a specified setting. Bartlett, Long, Lugosi, and Tsigler characterized when the minimum-norm interpolating solution in high-dimensional linear regression with quadratic loss can achieve near-optimal prediction accuracy.^[26] Their conditions use effective-rank properties of the feature covariance and require many directions that are unimportant for prediction relative to the sample size.

This result is narrower than the slogan that noise "averages out in high dimensions." It concerns a particular linear model, loss, interpolating rule, and covariance structure. Even within that framework, the spectrum matters. It does not establish that arbitrary neural networks, datasets, optimizers, or forms of label noise overfit benignly.

The term is useful because it separates interpolation from harmful generalization error. A fitted model can have zero empirical risk while its excess population risk remains small. Whether that occurs must be demonstrated or derived for the actual procedure.

Grokking

Grokking is delayed generalization after a model has already fit its training data. Power and colleagues introduced the term in experiments with small algorithmically generated binary-operation tables. In some runs, validation accuracy remained near chance long after training accuracy became perfect and then rose sharply with extended optimization.^[27] Their paper reported that weight decay was particularly effective in the tasks studied, not that weight decay is necessary for every instance of grokking.

Nanda, Lawrence Chan, Tom Lieberum, Jess Smith, and Jacob Steinhardt used mechanistic interpretability to analyze a one-layer transformer trained on modular addition.^[28] They identified a learned Fourier-based algorithm and described three phases: memorization, circuit formation, and cleanup. In that experiment, the sudden test-accuracy transition occurred after the generalizing mechanism had formed, as weight decay removed memorizing components. This is a mechanistic result for the analyzed model and task, not a universal circuit description.

Subsequent research has broadened the picture. Pascal Jr Tikeng Notsawo, Guillaume Dumas, and Guillaume Rabusseau reported theoretical and empirical settings in which L1 or domain-specific regularization can replace L2 regularization, and settings in which depth or data structure supplies implicit regularization without an explicit L2 penalty.^[29] Grokking therefore does not support a general recommendation to ignore early stopping. It is an experimentally contingent phenomenon whose occurrence, delay, and mechanism depend on data, architecture, optimization, and regularization.

How the modern phenomena relate

Scroll sideways for more →

None of these rows can be inferred from training loss alone.

Domain-specific considerations

Reinforcement learning

In reinforcement learning, the independent unit may be an environment layout, initial state, seed, task, or transition regime rather than an individual observation. If the same fixed environments are used for both development and reporting, an agent can specialize to them. Cobbe and colleagues introduced procedurally generated CoinRun levels to create distinct training and test environment sets and documented overfitting in that benchmark, including with surprisingly large training sets in their experiments.^[30]

Their reported interventions included deeper convolutional architectures and methods such as L2 regularization, dropout, data augmentation, and batch normalization. Those findings are benchmark-specific. A reinforcement-learning evaluation should state how environments, seeds, reward functions, and stochastic transitions are separated, and whether the claim concerns new instances of the same generator or transfer to a different generator.

Privacy

Overfitting can increase privacy risk because a model may behave differently on training members and nonmembers. Yeom and colleagues analyzed membership inference attacks and showed, under their attack and assumptions, a relationship between overfitting and membership advantage.^[31] They also showed that overfitting is not necessary for membership or attribute inference. Robustness properties, influential records, model outputs, and malicious training can create risk even without a conventional generalization gap.

Reducing the gap is therefore not a privacy guarantee. Privacy assessment requires an explicit threat model, attack evaluation, access assumptions, and, where appropriate, formal mechanisms such as differential privacy.

Scientific and high-stakes prediction

In medical, social, and scientific applications, preprocessing and split design often dominate the apparent model comparison. Measurements from one person or experimental unit must not be scattered across folds when the target is a new unit. Temporal prediction must not use future-derived features. Site-specific artifacts can substitute for the biological or social signal of interest.^[7][9]

For high-stakes claims, a complete report should include the cohort construction, exclusions, missing-data treatment, label timing, preprocessing fit, candidate search, split unit, external validation, subgroup results, calibration, and uncertainty. A high cross-validation score without that information is not enough to rule out overfitting.

A practical audit

The following sequence avoids arbitrary accuracy thresholds.

1. Define the claim. Specify the outcome, loss or utility, target population, prediction time, sampling unit, and acceptable error.
2. Trace provenance. Identify duplicates, shared entities, future information, target-derived features, label sources, and preprocessing dependencies.
3. Freeze split roles. Separate parameter fitting, model selection, and final assessment. Use grouped, temporal, or nested splitting when required.
4. Prespecify baselines. Include simple and domain-relevant comparators before expanding the search.
5. Fit the complete pipeline inside each training split. This includes imputation, normalization, feature selection, augmentation-policy fitting, and calibration choices.
6. Inspect absolute performance and gaps. Compare training with development performance, but also report whether either is useful.
7. Quantify variability. Refit across justified seeds or resamples, show intervals, and report sensitivity to influential observations and split choices.
8. Test shortcuts and shift. Use subgroup, temporal, site, corruption, ablation, and negative-control tests that match plausible failures.
9. Choose a mitigation based on evidence. Data correction, penalization, augmentation, early stopping, ensembling, or capacity changes solve different problems.
10. Assess once on locked data. Do not feed the result back into development while continuing to call it a final test.
11. Document the search. Report candidate families, hyperparameter ranges, stopping rules, selected checkpoint, and unsuccessful alternatives relevant to selection bias.
12. Monitor after deployment. New shift, feedback loops, and label changes can invalidate an earlier generalization estimate without retroactively proving the original model was overfit.

The desired outcome is not necessarily the smallest train-test gap. It is low, well-estimated risk for the intended use. A deliberately regularized model may have higher training loss and lower target risk. An underfit model may have a tiny gap and unacceptable target risk. A calibrated audit keeps those cases distinct.

Common misconceptions

• "A model with more parameters than examples is overfit." Parameter count alone does not determine effective capacity or population risk. Overparameterized interpolating models can generalize, while small models can overfit an adaptive feature search.^[23][24][26]
• "Perfect training accuracy proves memorization." It proves correct fit under the training metric. Whether the learned rule is a lookup, a general algorithm, or a mixture requires other evidence.^[27][28]
• "A large train-validation gap proves the model learned noise." The gap warrants investigation, but sampling variation, leakage, metric mismatch, dependence, or shift can also contribute.
• "A small gap proves good generalization." Both scores may be poor, the holdout may be contaminated, or both samples may omit the deployment population.
• "Cross-validation prevents overfitting." It estimates a procedure under a split design. Reusing the same folds to search and report the winner can overfit the cross-validation criterion.^[6][7]
• "More data always help." More representative independent information usually helps, but duplicated, biased, incorrectly labeled, or near-threshold data can have different effects. Sample-wise double descent has also been observed in particular regimes.^[25]
• "Weight decay is the same as L2 regularization." They coincide only under particular optimization updates. Adaptive methods motivate decoupled weight decay.^[20]
• "Batch normalization, pruning, or dropout automatically prevents overfitting." Each is a procedure whose effect depends on the architecture, data, optimizer, and evaluation.
• "Modern double descent means early stopping is obsolete." It means the best stopping rule is empirical and task-dependent. Early stopping remains useful in many settings.^[14][15][25]
• "Grokking requires weight decay." Weight decay was effective in the original experiments, but later work documents explicit and implicit alternatives.^[27][29]

Related concepts

Overfitting is one member of a broader set of model-development failures:

• Underfitting: the fitted procedure lacks the representation, optimization, or information needed to perform well even on development data.
• Distribution shift: the target distribution differs from the development distribution.
• Data leakage: unavailable or held-out information influences development or evaluation.
• Selection bias: the reported result is favored by the same noisy criterion used to choose it.
• Miscalibration: predicted probabilities do not match observed frequencies, even if ranking performance is acceptable.
• Shortcut learning: the model uses a predictive but unintended feature that may fail under a relevant change.
• Memorization: the predictor retains or reproduces training-specific information; this can contribute to overfitting or privacy risk but is not identical to either.

See also: Machine learning, supervised learning, deep learning, and machine learning terms.

References

1. ^Goodfellow, I., Bengio, Y., and Courville, A. (2016). *Deep Learning*, Chapter 5, Machine Learning Basics. MIT Press. deeplearningbook.org/...ml
2. ^Shalev-Shwartz, S., and Ben-David, S. (2014). *Understanding Machine Learning: From Theory to Algorithms*. Cambridge University Press. doi.org/...CBO9781107298019
3. ^Hastie, T., Tibshirani, R., and Friedman, J. (2009). *The Elements of Statistical Learning: Data Mining, Inference, and Prediction*, second edition. Springer. hastie.su.domains/ElemStatLearn
4. ^Geman, S., Bienenstock, E., and Doursat, R. (1992). "Neural Networks and the Bias/Variance Dilemma." *Neural Computation*, 4(1), 1-58. doi.org/...neco.1992.4.1.1
5. ^Bousquet, O., and Elisseeff, A. (2002). "Stability and Generalization." *Journal of Machine Learning Research*, 2, 499-526. jmlr.org/...bousquet02a
6. ^Cawley, G. C., and Talbot, N. L. C. (2010). "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation." *Journal of Machine Learning Research*, 11, 2079-2107. jmlr.org/...cawley10a
7. ^Varma, S., and Simon, R. (2006). "Bias in Error Estimation When Using Cross-validation for Model Selection." *BMC Bioinformatics*, 7, 91. doi.org/...1471-2105-7-91
8. ^Dwork, C., Feldman, V., Hardt, M., Pitassi, T., Reingold, O., and Roth, A. (2015). "The Reusable Holdout: Preserving Validity in Adaptive Data Analysis." *Science*, 349(6248), 636-638. doi.org/...science.aaa9375
9. ^Kapoor, S., and Narayanan, A. (2023). "Leakage and the Reproducibility Crisis in Machine-learning-based Science." *Patterns*, 4(9), 100804. doi.org/...j.patter.2023.100804
10. ^Moreno-Torres, J. G., Raeder, T., Alaiz-Rodriguez, R., Chawla, N. V., and Herrera, F. (2012). "A Unifying View on Dataset Shift in Classification." *Pattern Recognition*, 45(1), 521-530. doi.org/...j.patcog.2011.06.019
11. ^Tibshirani, R. (1996). "Regression Shrinkage and Selection via the Lasso." *Journal of the Royal Statistical Society: Series B*, 58(1), 267-288. doi.org/...j.2517-6161.1996.tb02080.x
12. ^Zou, H., and Hastie, T. (2005). "Regularization and Variable Selection via the Elastic Net." *Journal of the Royal Statistical Society: Series B*, 67(2), 301-320. doi.org/...j.1467-9868.2005.00503.x
13. ^Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). "Dropout: A Simple Way to Prevent Neural Networks from Overfitting." *Journal of Machine Learning Research*, 15, 1929-1958. jmlr.org/...srivastava14a
14. ^Prechelt, L. (1998). "Early Stopping - But When?" In G. B. Orr and K.-R. Muller, editors, *Neural Networks: Tricks of the Trade*. Springer. doi.org/...3-540-49430-8_3
15. ^Yao, Y., Rosasco, L., and Caponnetto, A. (2007). "On Early Stopping in Gradient Descent Learning." *Constructive Approximation*, 26, 289-315. doi.org/...s00365-006-0663-2
16. ^Hardt, M., Recht, B., and Singer, Y. (2016). "Train Faster, Generalize Better: Stability of Stochastic Gradient Descent." *Proceedings of the 33rd International Conference on Machine Learning*, PMLR 48, 1225-1234. proceedings.mlr.press/...hardt16
17. ^Bishop, C. M. (1995). "Training with Noise is Equivalent to Tikhonov Regularization." *Neural Computation*, 7(1), 108-116. doi.org/...neco.1995.7.1.108
18. ^Breiman, L. (1996). "Bagging Predictors." *Machine Learning*, 24, 123-140. doi.org/...BF00058655
19. ^Breiman, L. (2001). "Random Forests." *Machine Learning*, 45, 5-32. doi.org/...A:1010933404324
20. ^Loshchilov, I., and Hutter, F. (2019). "Decoupled Weight Decay Regularization." *International Conference on Learning Representations*. arxiv.org/...1711.05101
21. ^Zhang, H., Cisse, M., Dauphin, Y. N., and Lopez-Paz, D. (2018). "mixup: Beyond Empirical Risk Minimization." *International Conference on Learning Representations*. arxiv.org/...1710.09412
22. ^Tahmasebi, B., Weber, M., and Jegelka, S. (2026). "Data Augmentation: A Fourier Analysis Perspective." *Proceedings of Thirty Ninth Conference on Learning Theory*, PMLR 336, 6114-6155. proceedings.mlr.press/...tahmasebi26a
23. ^Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2017). "Understanding Deep Learning Requires Rethinking Generalization." *International Conference on Learning Representations*. arxiv.org/...1611.03530
24. ^Belkin, M., Hsu, D., Ma, S., and Mandal, S. (2019). "Reconciling Modern Machine-Learning Practice and the Classical Bias-Variance Trade-off." *Proceedings of the National Academy of Sciences*, 116(32), 15849-15854. doi.org/...pnas.1903070116
25. ^Nakkiran, P., Kaplun, G., Bansal, Y., Yang, T., Barak, B., and Sutskever, I. (2020). "Deep Double Descent: Where Bigger Models and More Data Hurt." *International Conference on Learning Representations*. arxiv.org/...1912.02292
26. ^Bartlett, P. L., Long, P. M., Lugosi, G., and Tsigler, A. (2020). "Benign Overfitting in Linear Regression." *Proceedings of the National Academy of Sciences*, 117(48), 30063-30070. doi.org/...pnas.1907378117
27. ^Power, A., Burda, Y., Edwards, H., Babuschkin, I., and Misra, V. (2022). "Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets." arXiv. arxiv.org/...2201.02177
28. ^Nanda, N., Chan, L., Lieberum, T., Smith, J., and Steinhardt, J. (2023). "Progress Measures for Grokking via Mechanistic Interpretability." *International Conference on Learning Representations*. openreview.net/forum
29. ^Tikeng Notsawo, P. Jr, Dumas, G., and Rabusseau, G. (2025). "Grokking Beyond the Euclidean Norm of Model Parameters." *Proceedings of the 42nd International Conference on Machine Learning*, PMLR 267, 28552-28618. proceedings.mlr.press/...junior25a
30. ^Cobbe, K., Klimov, O., Hesse, C., Kim, T., and Schulman, J. (2019). "Quantifying Generalization in Reinforcement Learning." *Proceedings of the 36th International Conference on Machine Learning*, PMLR 97, 1282-1289. proceedings.mlr.press/...cobbe19a
31. ^Yeom, S., Giacomelli, I., Fredrikson, M., and Jha, S. (2018). "Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting." *31st IEEE Computer Security Foundations Symposium*, 268-282. doi.org/...CSF.2018.00027