# Model selection

> Source: https://aiwiki.ai/wiki/model_selection
> Updated: 2026-11-18
> Fact-checked: 2026-11-18
> Categories: Machine Learning, Model Evaluation, Statistics
> License: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/) - attribute to "AI Wiki (aiwiki.ai)"
> Cite as: AI Wiki. "Model selection." aiwiki.ai, 18 Nov 2026. https://aiwiki.ai/wiki/model_selection
> From AI Wiki (https://aiwiki.ai), the free encyclopedia of artificial intelligence. Reuse freely with attribution.

**Model selection** is the task of choosing, from a set of candidate statistical or [machine learning](https://aiwiki.ai/wiki/machine_learning) models, the one to use for a given problem, based on how well each candidate is expected to perform on new data. The candidates can be different model families, different subsets of input variables, or the same learning method run with different [hyperparameter](https://aiwiki.ai/wiki/hyperparameter) values. In *The Elements of Statistical Learning*, Trevor Hastie, Robert Tibshirani and Jerome Friedman describe model selection as comparing the estimated performance of several models so that one can be chosen, and treat it as a separate goal from **model assessment**, which estimates how large the prediction error of the finally chosen model will be on new data.[1]

The central difficulty is that a model's error on the data it was fitted to is a poor guide to its error on new data: more flexible models fit the training sample more closely, but they can end up fitting its noise.[1] Model selection methods therefore estimate out-of-sample error in one of two ways. They either reuse the data, as in [cross-validation](https://aiwiki.ai/wiki/cross-validation) and the bootstrap, or they correct the training fit analytically with a penalty for complexity, as in the Akaike information criterion (AIC), the Bayesian information criterion (BIC), minimum description length (MDL) and [structural risk minimization](https://aiwiki.ai/wiki/structural_risk_minimization_srm).[1] Bayesian statistics adds a third route, comparing models by their posterior probabilities or averaging over them instead of picking one.[1][2]

## Selection and assessment

Hastie, Tibshirani and Friedman describe two separate goals. Model selection compares candidates to pick one; model assessment, done once a model has been picked, estimates how well it will generalize.[1] When data are plentiful, they recommend splitting the dataset at random into three parts: a [training set](https://aiwiki.ai/wiki/training_set) to fit the models, a [validation set](https://aiwiki.ai/wiki/validation_set) to estimate prediction error for model selection, and a [test set](https://aiwiki.ai/wiki/test_set) to assess the final model. They suggest a typical split of 50% for training and 25% each for validation and testing, while noting that no general rule fits every problem.[1] The test set should be used only at the end of the analysis. If it is consulted repeatedly and the model with the lowest test error is chosen, the reported test error of that model will underestimate its true error, sometimes substantially.[1]

Most of the methods described below exist for the common case in which there is not enough data for a three-way split. They approximate the validation step either analytically or by reusing the sample.[1]

Statisticians also distinguish two aims within model selection itself. When the goal is estimation or prediction, a procedure is judged by how close its chosen model's risk comes to that of the best candidate, a property called efficiency. When the goal is identification, it is assumed that one candidate is the true model, and a procedure is judged by whether the probability of choosing that model tends to one as the sample grows, a property called consistency.[3] In many settings no candidate is exactly true, and efficiency is then the only well-defined property.[3]

## Bias, variance and complexity

Model selection is closely tied to the [bias-variance tradeoff](https://aiwiki.ai/wiki/bias_variance_tradeoff). If $Y = f(X) + \varepsilon$ with $E(\varepsilon) = 0$ and $\operatorname{Var}(\varepsilon) = \sigma_\varepsilon^2$, the expected squared prediction error of a fitted model $\hat{f}$ at an input point $x_0$ splits into three parts:[1]

$$\operatorname{Err}(x_0) = \sigma_\varepsilon^2 + \left[E\hat{f}(x_0) - f(x_0)\right]^2 + E\left[\hat{f}(x_0) - E\hat{f}(x_0)\right]^2$$

The first term is irreducible noise, the second is the squared bias and the third is the variance. Making a model more complex typically lowers the bias and raises the variance, so some intermediate complexity gives the lowest expected test error.[1] Training error does not reveal where that point lies: it keeps falling as complexity grows, often down to zero, and a model with zero training error is usually [overfit](https://aiwiki.ai/wiki/overfitting) and generalizes poorly.[1] In a 1992 paper in *Neural Computation*, Stuart Geman, Elie Bienenstock and René Doursat treated feedforward [neural networks](https://aiwiki.ai/wiki/neural_network) as nonparametric regression estimators and used this statistical viewpoint, the "bias/variance dilemma" of their title, to discuss the strengths and weaknesses of neural models.[4]

## Validation and cross-validation

### Holdout validation

The simplest approach fits each candidate on one part of the data and compares their errors on a separate held-out part (see [holdout data](https://aiwiki.ai/wiki/holdout_data)). Sebastian Raschka's 2018 review of model evaluation and selection describes the holdout method as common but does not recommend it for small datasets.[5]

### Cross-validation

[Cross-validation](https://aiwiki.ai/wiki/cross-validation) reuses the sample instead of setting part of it aside for good. Mervyn Stone's 1974 paper in the *Journal of the Royal Statistical Society, Series B* applied a general form of the cross-validation criterion to the choice and assessment of predictions in univariate estimation, linear regression and analysis of variance,[6] and Seymour Geisser described a related "predictive sample reuse" method in the *Journal of the American Statistical Association* in 1975.[7]

In $K$-fold cross-validation the data are split into $K$ roughly equal parts. For each part $k$, the model is fitted on the other $K - 1$ parts and its prediction error is measured on part $k$; the $K$ estimates are then combined. For a family of models indexed by a tuning parameter $\alpha$, the cross-validation estimate is

$$\operatorname{CV}(\hat{f}, \alpha) = \frac{1}{N} \sum_{i=1}^{N} L\left(y_i, \hat{f}^{-\kappa(i)}(x_i, \alpha)\right)$$

where $L$ is the [loss function](https://aiwiki.ai/wiki/loss_function), $N$ is the number of observations and $\hat{f}^{-\kappa(i)}$ is the model fitted without the fold that contains observation $i$. The chosen value $\hat{\alpha}$ minimizes this curve, and the final model is then refitted to all the data.[1] The case $K = N$ is leave-one-out cross-validation, which is approximately unbiased for the expected prediction error but can have high variance because the $N$ training sets are so similar; Hastie and colleagues recommend five- or tenfold cross-validation as a compromise.[1] Ron Kohavi's study at the Fourteenth International Joint Conference on Artificial Intelligence in 1995, based on more than half a million runs of the C4.5 and naive Bayes algorithms on real-world datasets, concluded that ten-fold stratified cross-validation was the best method for model selection on datasets like those he tested, even when more folds were affordable.[8]

Hastie and colleagues also describe two related techniques. Under the "one-standard-error" rule, the analyst picks the simplest model whose cross-validated error is within one standard error of the best model's error.[1] For linear smoothers, generalized cross-validation (GCV) approximates leave-one-out cross-validation using the trace of the smoother matrix, which can be cheaper to compute.[1]

### Doing cross-validation correctly

Cross-validation estimates the error of a whole modeling procedure, so every step that looks at the labels has to be repeated inside each fold. Hastie, Tibshirani and Friedman give a simulated example with 50 samples in two equal classes and 5,000 predictors that are pure noise, so that any classifier's true error rate is 50%. Screening for the 100 predictors most correlated with the labels on the full dataset, and only then cross-validating a nearest-neighbor classifier on those predictors, gave an average cross-validated error of 3% over 50 simulations. The correct procedure repeats the screening inside each fold, using only the samples outside that fold, and the predictors it selects then show no spurious correlation with the held-out labels. Unsupervised screening that ignores the labels, such as keeping the highest-variance predictors, can safely be done beforehand.[1] The same problem, a form of [data leakage](https://aiwiki.ai/wiki/data_leakage), arises with hyperparameter tuning. Sudhir Varma and Richard Simon showed in 2006 that the cross-validated error of a classifier whose parameters had themselves been tuned by cross-validation was a substantially biased estimate of its true error. On simulated "null" datasets with no real class difference, the tuned support vector machine had an estimated error below 30% on 38% of datasets, while its performance on independent test data was no better than chance. A nested procedure, with an inner loop for tuning and an outer loop for error estimation, gave an almost unbiased estimate.[9]

Gavin Cawley and Nicola Talbot argued in the *Journal of Machine Learning Research* in 2010 that the variance of a selection criterion matters at least as much as its bias. Optimizing a noisy criterion such as a $k$-fold cross-validation score over many candidates can overfit the criterion itself, and they found that the resulting loss in performance was often comparable in size to the differences between learning algorithms.[10]

## Information criteria

Information criteria replace held-out data with a penalty added to a goodness-of-fit term, usually the maximized log-likelihood. Each candidate model receives a score, and the model with the smallest score is chosen.[1] In the formulas below, $\hat{L}$ is the maximized likelihood, $k$ is the number of estimated parameters and $n$ is the sample size; in $C_p$, $\overline{\text{err}}$ is the average training error under squared-error loss and $\hat{\sigma}_\varepsilon^2$ is an estimate of the noise variance.

| Criterion | Key reference | Form | Main motivation |
|---|---|---|---|
| Mallows' $C_p$ | C. L. Mallows, 1973 | $\overline{\text{err}} + \frac{2k}{n}\hat{\sigma}_\varepsilon^2$ | Estimate of in-sample prediction error for least squares[1][11] |
| AIC | Hirotugu Akaike, 1973 and 1974 | $-2\ln\hat{L} + 2k$ | Estimate of expected Kullback-Leibler information loss[12][13] |
| AICc | Sugiura, 1978; Hurvich and Tsai, 1989 | $-2\ln\hat{L} + \frac{2kn}{n - k - 1}$ | Small-sample bias correction of AIC[14][15] |
| BIC (Schwarz criterion) | Gideon Schwarz, 1978 | $-2\ln\hat{L} + k\ln n$ | Large-sample approximation to Bayesian model choice[16][1] |
| Hannan-Quinn (HQIC) | Hannan and Quinn, 1979 | $-2\ln\hat{L} + 2k\ln\ln n$ | Strongly consistent order selection for autoregressions[17][15] |
| MDL (two-part) | Jorma Rissanen, 1978 | $-\ln P(x^n \mid \hat{\theta}) + \frac{k}{2}\ln n$ | Shortest description of the data[18][19] |
| DIC | Spiegelhalter, Best, Carlin and van der Linde, 2002 | $\bar{D} + p_D$ | Bayesian fit plus effective number of parameters[20] |
| WAIC | Sumio Watanabe, 2010 | (see text) | Estimate of Bayes generalization loss, valid for singular models[21] |

### Akaike information criterion

Hirotugu Akaike proposed the criterion in a 1973 paper for the Second International Symposium on Information Theory,[1][13] and set out its use for statistical model identification in "A new look at the statistical model identification", published in *IEEE Transactions on Automatic Control* in December 1974. In that paper he argued that classical hypothesis testing was not an adequate procedure for identifying a statistical model, and proposed choosing among competing models the one that minimizes $\text{AIC} = (-2)\log(\text{maximum likelihood}) + 2(\text{number of independently adjusted parameters})$, a procedure he called the minimum information theoretical criterion (AIC) estimate, or MAICE.[12] Kenneth Burnham and David Anderson explain the theory behind the formula: the maximized log-likelihood is a biased estimate of the expected log-likelihood of the fitted model on new data, and the bias is approximately equal to $k$, so the penalty is a bias correction rather than an arbitrary charge. AIC is therefore an approximately unbiased estimator, for large samples and good models, of the expected relative [Kullback-Leibler](https://aiwiki.ai/wiki/kl_divergence) information lost when the fitted model is used to approximate the true process.[13]

For a Gaussian model with known noise variance, AIC is equivalent to Mallows' $C_p$, and Hastie and colleagues treat the two together.[1] For linear fitting methods that use shrinkage, such as [ridge regression](https://aiwiki.ai/wiki/ridge_regression) and smoothing splines, the parameter count in $C_p$ can be replaced by the effective number of parameters, which for a linear smoother $\hat{y} = Sy$ is $\operatorname{trace}(S)$.[1]

When $k$ is large relative to $n$, AIC is biased. Clifford Hurvich and Chih-Ling Tsai derived a corrected version, AICc, for regression and autoregressive time series models in *Biometrika* in 1989, intended for small samples or for cases where the number of parameters is a moderate to large fraction of the sample size.[14] Burnham and Anderson also credit the corrected criterion to Nariaki Sugiura's 1978 work.[13] Burnham and Anderson advise using AICc unless $n/k$ is greater than about 40 for the largest model, and argue that many reports of AIC overfitting come from using AIC where AICc was needed.[13] Because only differences in AIC between models are meaningful, they rescale each model's score as $\Delta_i = \text{AIC}_i - \text{AIC}_{\min}$. As rough guidelines, models with $\Delta_i \le 2$ have substantial support, those with $\Delta_i$ between 4 and 7 have considerably less, and those with $\Delta_i > 10$ have essentially none.[13] The normalized quantities $w_i = \exp(-\Delta_i/2) / \sum_r \exp(-\Delta_r/2)$, called Akaike weights, can be read as the weight of evidence that model $i$ is the best approximating model in the set.[13]

### Bayesian information criterion

Gideon Schwarz's 1978 paper "Estimating the Dimension of a Model" in *The Annals of Statistics* treated the choice among models of different dimensions by finding its Bayes solution and evaluating the leading terms of its asymptotic expansion. Those terms do not depend on the prior distribution, which Schwarz argued makes them a valid large-sample criterion outside the Bayesian setting too.[16] The resulting criterion, $\text{BIC} = -2\log\hat{L} + k\log n$, has the same form as AIC with the factor 2 replaced by $\log n$. When $n$ is greater than $e^2 \approx 7.4$, BIC penalizes complexity more heavily and so tends to choose simpler models.[1]

Choosing the model with the smallest BIC amounts to choosing the model with the largest approximate posterior probability under equal prior probabilities for the models, and the BIC values of a set of $M$ models can be turned into estimated posterior model probabilities:[1]

$$\Pr(\mathcal{M}_m \mid Z) \approx \frac{e^{-\frac{1}{2}\text{BIC}_m}}{\sum_{\ell=1}^{M} e^{-\frac{1}{2}\text{BIC}_\ell}}$$

Robert Kass and Adrian Raftery's review of Bayes factors describes BIC as a rough approximation to the logarithm of the Bayes factor that is easy to use and needs no prior distributions.[22]

### AIC versus BIC

The two criteria are built for different goals. BIC is consistent: if the true model is among the candidates, the probability that BIC selects it tends to one as the sample grows. AIC is not consistent and tends to choose models that are too complex as $n \to \infty$; on the other hand, in finite samples BIC often picks models that are too simple because of its heavier penalty. Hastie, Tibshirani and Friedman conclude that there is no clear choice between them.[1] In a simulation in their book covering regression and classification with nearest-neighbor and linear models, AIC worked well in all four scenarios, BIC did nearly as well, and structural risk minimization gave mixed results.[1]

Yuhong Yang showed in 2005 that, for regression, the strengths of the two cannot be fully combined. BIC is consistent in selecting the true model and AIC is minimax-rate optimal for estimating the regression function, and Yang proved that any consistent selection criterion must be suboptimal in that minimax sense, and that Bayesian model averaging cannot be minimax-rate optimal either.[23] Arlot and Celisse note that the question of whether one procedure can have both properties is often called the AIC-BIC dilemma.[3] Burnham and Anderson argue that the choice should not be framed as frequentist versus Bayesian, since AIC can be justified as Bayesian under a particular prior on models and BIC can be derived without Bayesian arguments. In their view the choice should depend on what is assumed about reality and on the purpose of the inference.[13]

### Relation to cross-validation

Information criteria and cross-validation are linked asymptotically. Stone showed in 1977 that, when models are fitted by maximum likelihood and predictions are scored by log-likelihood, choosing a model by cross-validation is asymptotically equivalent to choosing it by Akaike's criterion.[24] Jun Shao showed in 1993 that leave-one-out cross-validation, which is asymptotically equivalent to AIC, $C_p$ and the bootstrap, is inconsistent for selecting among linear models: the probability of selecting the model with the best predictive ability does not tend to one. Shao showed that consistency can be restored with leave-$n_v$-out cross-validation in which the fraction of observations held out for validation, $n_v/n$, tends to one, which he called "somewhat shocking" because it runs opposite to the usual leave-one-out recipe.[25] Arlot and Celisse's survey concludes that when the goal is to identify the true model, consistency often requires setting aside a large share of the data for validation, although in some frameworks larger training sets also give consistency.[3]

### Minimum description length

The minimum description length principle, developed mainly by Jorma Rissanen in a series of papers starting with his 1978 paper in *Automatica*,[18][19] treats model selection as data compression: the preferred model is the one that gives the shortest code for transmitting the data.[1][19] In that first paper Rissanen used a two-part code and showed that, asymptotically and under regularity conditions, the code length for a $k$-parameter model is $-\log P(x^n \mid \hat{\theta}) + \frac{k}{2}\log n$, ignoring constant terms. Schwarz derived the same expression from Bayesian arguments in the same year, and model selection by this formula became known as BIC.[19] Hastie, Tibshirani and Friedman note that the MDL approach gives a criterion formally identical to BIC, although motivated by optimal coding.[1] Peter Grunwald's tutorial on MDL notes that this form ignores constant terms that can be important and, because it does not take the functional form of the model into account, often does not work very well in practice. He also rejects the claim that MDL is the same as BIC, which he says is wrong even for Rissanen's 1989 version of MDL.[19]

### Criteria for Bayesian models

For hierarchical [Bayesian](https://aiwiki.ai/wiki/bayesian_statistics) models, where the number of parameters is not clearly defined, David Spiegelhalter, Nicola Best, Bradley Carlin and Angelika van der Linde proposed in 2002 an effective number of parameters $p_D$, defined as the posterior mean of the deviance minus the deviance at the posterior means of the parameters. Adding $p_D$ to the posterior mean deviance gives the deviance information criterion (DIC), which is easy to compute from [Markov chain Monte Carlo](https://aiwiki.ai/wiki/markov_chain_monte_carlo) output.[20]

Sumio Watanabe proposed the widely applicable information criterion (WAIC) within his singular learning theory, noting that many learning machines are singular statistical models, for which the asymptotic behavior of cross-validation had been unknown, although in regular models leave-one-out cross-validation is asymptotically equivalent to AIC.[21] The expected value of WAIC is asymptotically equal to the average Bayes generalization loss. In a 2010 *Journal of Machine Learning Research* paper, Watanabe proved that the Bayes cross-validation loss and WAIC are asymptotically equivalent, so model selection and hyperparameter optimization with either give asymptotically the same result, and he showed that DIC differs from both.[21] Aki Vehtari, Andrew Gelman and Jonah Gabry later gave fast computations of leave-one-out cross-validation and WAIC from existing posterior draws, using Pareto-smoothed importance sampling (PSIS), and reported that PSIS-LOO is more robust than WAIC in finite samples with weak priors or influential observations. They implemented the methods in the R package loo.[26]

## Bayesian model comparison and averaging

In the Bayesian approach, each candidate model $\mathcal{M}_m$ receives a posterior probability proportional to its prior probability times its marginal likelihood $\Pr(Z \mid \mathcal{M}_m)$, which integrates the likelihood over the model's prior on its parameters. The ratio of two models' marginal likelihoods is the Bayes factor, the contribution of the data to the posterior odds.[1] Kass and Raftery note that Bayes factors do not require the models being compared to be nested and can incorporate external information.[22]

Instead of choosing a single model, an analyst can average over several. Jennifer Hoeting, David Madigan, Adrian Raftery and Chris Volinsky argued in a 1999 tutorial that the usual practice of selecting one model and then proceeding as if it had generated the data ignores model uncertainty and leads to over-confident inferences. Bayesian model averaging weights each model's predictions by its posterior probability, and in their examples it improved out-of-sample predictive performance.[2] Burnham and Anderson also present methods of model averaging based on Akaike weights.[13]

## Selection through regularization

Some methods carry out part of the selection while fitting. The [lasso](https://aiwiki.ai/wiki/lasso_regression), proposed by Robert Tibshirani in 1996, minimizes the residual sum of squares subject to a bound on the sum of the absolute values of the coefficients. The constraint tends to set some coefficients exactly to zero, so the lasso performs variable selection, and Tibshirani's simulations suggested it combines the interpretability of subset selection with the stability of ridge regression.[27] The strength of such a penalty is itself a hyperparameter that has to be chosen, usually by cross-validation or an information criterion; scikit-learn, for example, provides estimators such as `LassoCV` that cross-validate along the whole regularization path, and `LassoLarsIC`, which picks the penalty with AIC or BIC.[28]

Vladimir Vapnik's structural risk minimization fits a nested sequence of models of increasing [VC dimension](https://aiwiki.ai/wiki/vc_dimension) and chooses the one with the smallest upper bound on the test error. Hastie, Tibshirani and Friedman note that such bounds are often loose but can still rank models, and that the main drawback is the difficulty of calculating the VC dimension of a class of functions.[1]

## Hyperparameter search

In machine learning practice, model selection often means [hyperparameter tuning](https://aiwiki.ai/wiki/hyperparameter_tuning): searching a space of settings for the configuration with the best validation or cross-validation score. The scikit-learn documentation describes a search as made of an estimator, a parameter space, a method for searching or sampling candidates, a cross-validation scheme and a score function. Its `GridSearchCV` tries every combination on a grid, `RandomizedSearchCV` samples a fixed number of candidates from given distributions, and successive-halving variants evaluate all candidates with a small budget (for example, a subset of the training samples) and give more resources only to the best performers in later rounds. For bagged ensembles such as random forests, the out-of-bag samples give an error estimate without a separate validation set.[28]

James Bergstra and [Yoshua Bengio](https://aiwiki.ai/wiki/yoshua_bengio) showed in 2012 that randomly chosen trials are more efficient than a grid. In their experiments, random search found neural network configurations as good as or better than grid search within a small fraction of the computation time. Their explanation was that for most datasets only a few hyperparameters matter, and which ones matter differs from dataset to dataset.[29] [Bayesian optimization](https://aiwiki.ai/wiki/bayesian_optimization) treats the validation performance as an unknown function, models it with a [Gaussian process](https://aiwiki.ai/wiki/gaussian_process) and uses the posterior to choose the next setting to try; Jasper Snoek, Hugo Larochelle and Ryan Adams reported in 2012 that this approach could match or exceed expert-level tuning on algorithms including latent Dirichlet allocation, structured support vector machines and convolutional neural networks.[30] Auto-WEKA, described by Chris Thornton, Frank Hutter, Holger Hoos and Kevin Leyton-Brown in 2012, extended the search to choosing the learning algorithm and its hyperparameters together, using Bayesian optimization over all the classification methods in the WEKA toolkit.[31]

## Instability and inference after selection

Selection procedures can be unstable. Leo Breiman showed in 1996 that when a small change in the data can cause a large change in the sequence of fitted models, the gap between the error of the model chosen by test sets or cross-validation and that of the best model in the sequence is large. He proposed stabilizing such procedures by perturbing the data and averaging over many fitted sequences.[32]

Selection also affects statistical inference. Richard Berk and colleagues pointed out in 2013 that tests and confidence intervals computed after data-driven variable selection lose the guarantees classical theory provides for a model chosen in advance. They proposed valid "post-selection inference" by widening the intervals to hold simultaneously over all possible submodels, which makes them valid for any selection procedure but generally conservative.[33]

## Model selection in modern machine learning

Very large models complicate the classical picture of a U-shaped test-error curve. Mikhail Belkin, Daniel Hsu, Siyuan Ma and Soumik Mandal observed in 2019 that rich models such as neural networks are often trained to fit their training data exactly, which classical analysis would call overfitting, yet they still reach high test accuracy. They proposed a "double descent" curve in which test error falls again once model capacity grows past the point where the model can interpolate the training data, and reported evidence for it across a range of models and datasets.[34] The [double descent](https://aiwiki.ai/wiki/double_descent) article covers later work on the phenomenon in [deep learning](https://aiwiki.ai/wiki/deep_learning).

## Software

Widely used libraries implement these methods. [scikit-learn](https://aiwiki.ai/wiki/scikit_learn) groups its cross-validation splitters, search classes and validation-curve utilities under a "Model selection and evaluation" section of its user guide.[28] The Python package statsmodels provides functions for AIC, AICc, BIC and the Hannan-Quinn criterion.[15]

## See also

- [Cross-validation](https://aiwiki.ai/wiki/cross-validation)
- [Bias-variance tradeoff](https://aiwiki.ai/wiki/bias_variance_tradeoff)
- [Overfitting](https://aiwiki.ai/wiki/overfitting)
- [Hyperparameter tuning](https://aiwiki.ai/wiki/hyperparameter_tuning)
- [Model evaluation](https://aiwiki.ai/wiki/model_evaluation)
- [Regularization](https://aiwiki.ai/wiki/regularization)
- [Feature selection](https://aiwiki.ai/wiki/feature_selection)
- [Ensemble learning](https://aiwiki.ai/wiki/ensemble_learning)
- [AutoML](https://aiwiki.ai/wiki/automl)

## References

1. Hastie, Trevor; Tibshirani, Robert; Friedman, Jerome. "The Elements of Statistical Learning: Data Mining, Inference, and Prediction" (2nd ed.), chapter 7, "Model Assessment and Selection". Springer Series in Statistics, 2009. https://hastie.su.domains/ElemStatLearn/
2. Hoeting, Jennifer A.; Madigan, David; Raftery, Adrian E.; Volinsky, Chris T. "Bayesian model averaging: a tutorial". Statistical Science 14(4): 382-417, 1999. https://doi.org/10.1214/ss/1009212519
3. Arlot, Sylvain; Celisse, Alain. "A survey of cross-validation procedures for model selection". Statistics Surveys 4: 40-79, 2010. https://doi.org/10.1214/09-SS054
4. Geman, Stuart; Bienenstock, Elie; Doursat, Rene. "Neural Networks and the Bias/Variance Dilemma". Neural Computation 4(1): 1-58, 1992. https://doi.org/10.1162/neco.1992.4.1.1
5. Raschka, Sebastian. "Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning". arXiv:1811.12808, 2018. https://arxiv.org/abs/1811.12808
6. Stone, M. "Cross-Validatory Choice and Assessment of Statistical Predictions". Journal of the Royal Statistical Society, Series B 36(2): 111-133, 1974. https://doi.org/10.1111/j.2517-6161.1974.tb00994.x
7. Geisser, Seymour. "The Predictive Sample Reuse Method with Applications". Journal of the American Statistical Association 70(350): 320-328, 1975. https://doi.org/10.1080/01621459.1975.10479865
8. Kohavi, Ron. "A Study of Cross-Validation and Bootstrap for Accuracy Estimation and Model Selection". Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence (IJCAI), 1995. https://www.ijcai.org/Proceedings/95-2/Papers/016.pdf
9. Varma, Sudhir; Simon, Richard. "Bias in error estimation when using cross-validation for model selection". BMC Bioinformatics 7: 91, 2006. https://doi.org/10.1186/1471-2105-7-91
10. Cawley, Gavin C.; Talbot, Nicola L. C. "On Over-fitting in Model Selection and Subsequent Selection Bias in Performance Evaluation". Journal of Machine Learning Research 11: 2079-2107, 2010. https://jmlr.org/papers/v11/cawley10a.html
11. Mallows, C. L. "Some Comments on Cp". Technometrics 15(4): 661-675, 1973. https://doi.org/10.1080/00401706.1973.10489103
12. Akaike, H. "A new look at the statistical model identification". IEEE Transactions on Automatic Control 19(6): 716-723, December 1974. https://doi.org/10.1109/TAC.1974.1100705
13. Burnham, Kenneth P.; Anderson, David R. "Multimodel Inference: Understanding AIC and BIC in Model Selection". Sociological Methods & Research 33(2): 261-304, 2004. https://doi.org/10.1177/0049124104268644
14. Hurvich, Clifford M.; Tsai, Chih-Ling. "Regression and time series model selection in small samples". Biometrika 76(2): 297-307, 1989. https://doi.org/10.1093/biomet/76.2.297
15. statsmodels developers. "statsmodels/tools/eval_measures.py" (aic, aicc, bic, hqic). statsmodels source code, GitHub, accessed November 2026. https://github.com/statsmodels/statsmodels/blob/main/statsmodels/tools/eval_measures.py
16. Schwarz, Gideon. "Estimating the Dimension of a Model". The Annals of Statistics 6(2): 461-464, 1978. https://doi.org/10.1214/aos/1176344136
17. Hannan, E. J.; Quinn, B. G. "The Determination of the Order of an Autoregression". Journal of the Royal Statistical Society, Series B 41(2): 190-195, 1979. https://doi.org/10.1111/j.2517-6161.1979.tb01072.x
18. Rissanen, J. "Modeling by shortest data description". Automatica 14(5): 465-471, 1978. https://doi.org/10.1016/0005-1098(78)90005-5
19. Grunwald, Peter. "A tutorial introduction to the minimum description length principle". arXiv:math/0406077, 2004. https://arxiv.org/abs/math/0406077
20. Spiegelhalter, David J.; Best, Nicola G.; Carlin, Bradley P.; van der Linde, Angelika. "Bayesian Measures of Model Complexity and Fit". Journal of the Royal Statistical Society, Series B 64(4): 583-639, 2002. https://doi.org/10.1111/1467-9868.00353
21. Watanabe, Sumio. "Asymptotic Equivalence of Bayes Cross Validation and Widely Applicable Information Criterion in Singular Learning Theory". Journal of Machine Learning Research 11: 3571-3594, 2010. https://jmlr.org/papers/v11/watanabe10a.html
22. Kass, Robert E.; Raftery, Adrian E. "Bayes Factors". Journal of the American Statistical Association 90(430): 773-795, 1995. https://doi.org/10.1080/01621459.1995.10476572
23. Yang, Yuhong. "Can the strengths of AIC and BIC be shared? A conflict between model indentification and regression estimation". Biometrika 92(4): 937-950, 2005. https://doi.org/10.1093/biomet/92.4.937
24. Stone, M. "An Asymptotic Equivalence of Choice of Model by Cross-Validation and Akaike's Criterion". Journal of the Royal Statistical Society, Series B 39(1): 44-47, 1977. https://doi.org/10.1111/j.2517-6161.1977.tb01603.x
25. Shao, Jun. "Linear Model Selection by Cross-validation". Journal of the American Statistical Association 88(422): 486-494, 1993. https://doi.org/10.1080/01621459.1993.10476299
26. Vehtari, Aki; Gelman, Andrew; Gabry, Jonah. "Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC". Statistics and Computing 27(5): 1413-1432, 2017. https://doi.org/10.1007/s11222-016-9696-4
27. Tibshirani, Robert. "Regression Shrinkage and Selection Via the Lasso". Journal of the Royal Statistical Society, Series B 58(1): 267-288, 1996. https://doi.org/10.1111/j.2517-6161.1996.tb02080.x
28. scikit-learn developers. "3.2. Tuning the hyper-parameters of an estimator". scikit-learn 1.9.1 documentation, accessed November 2026. https://scikit-learn.org/stable/modules/grid_search.html
29. Bergstra, James; Bengio, Yoshua. "Random Search for Hyper-Parameter Optimization". Journal of Machine Learning Research 13: 281-305, 2012. https://jmlr.org/papers/v13/bergstra12a.html
30. Snoek, Jasper; Larochelle, Hugo; Adams, Ryan P. "Practical Bayesian Optimization of Machine Learning Algorithms". arXiv:1206.2944, 2012. https://arxiv.org/abs/1206.2944
31. Thornton, Chris; Hutter, Frank; Hoos, Holger H.; Leyton-Brown, Kevin. "Auto-WEKA: Combined Selection and Hyperparameter Optimization of Classification Algorithms". arXiv:1208.3719, 2012. https://arxiv.org/abs/1208.3719
32. Breiman, Leo. "Heuristics of instability and stabilization in model selection". The Annals of Statistics 24(6): 2350-2383, 1996. https://doi.org/10.1214/aos/1032181158
33. Berk, Richard; Brown, Lawrence; Buja, Andreas; Zhang, Kai; Zhao, Linda. "Valid post-selection inference". The Annals of Statistics 41(2): 802-837, 2013. https://doi.org/10.1214/12-AOS1077
34. Belkin, Mikhail; Hsu, Daniel; Ma, Siyuan; Mandal, Soumik. "Reconciling modern machine-learning practice and the classical bias-variance trade-off". Proceedings of the National Academy of Sciences 116(32): 15849-15854, 2019. https://doi.org/10.1073/pnas.1903070116

