Test loss

See also: Machine learning terms

Test loss is the value of a loss function computed on a held-out test data set: data that was used neither for training nor for validation or model selection. Because the test examples are untouched during development, the test loss serves as an approximately unbiased estimate of a model's generalization error, that is, how well it will perform on new data drawn from the same distribution. It is reported once, at the very end of a machine learning workflow, and the gap between it and the training loss (the "generalization gap") is a primary signal of overfitting. ^[1][2]

What is test loss?

Test loss is a metric that measures a model's loss against the test data set. The test data is a separate set of examples from the training data set and the validation data set. Running the model on the test set is the last step in a typical machine learning workflow, done after training and after all hyperparameter choices have already been made. The lower the test loss, the better the model has fit the kind of data it will see in the wild, assuming the test set was sampled from the same distribution as the data the model will face in production.

The number is calculated the same way as training loss or validation loss. You feed the test examples through the trained machine learning model, compute the loss function on each prediction versus its label, and average the per-example losses. What changes is what the number is used for. Training loss steers the optimizer through gradient descent. Validation loss drives model selection during development. Test loss does neither; it is the final number you write down and report.

Test loss is often described as an "unbiased estimate" of generalization performance, and the unbiasedness only holds if the test set was truly held out. The moment you start tweaking the model based on what the test set tells you, the test set has become a second validation set, and your test loss starts drifting optimistic. ^[1]

What is test loss trying to estimate?

Supervised learning frames its goal as minimizing expected risk: the average loss the model would incur if you could draw an infinite number of samples from the true data distribution. That expectation is impossible to compute, because the true distribution is unknown. The training loss is the empirical risk on the training set. The test loss is the empirical risk on a held-out sample not used for fitting or selection, which makes it a reasonable estimator of the true risk. ^[3]

The Wikipedia article on generalization error puts it cleanly: the test sample is "previously unseen by the algorithm and so represents a random sample from the joint probability distribution." ^[2] Test loss is a Monte Carlo estimate of generalization error, with variance that shrinks roughly with the square root of the test set size. A 100-example test set gives you a noisy estimate; a 10,000-example one is much tighter.

In empirical risk minimization, training loss is what the learner explicitly minimizes, while test loss is a sanity check that the minimization actually generalized. The difference between them, often called the generalization gap, is one of the central objects of study in statistical learning theory. ^[3]

How is test loss computed?

The computation depends on the problem and the loss function you picked.

Scroll sideways for more →

The convention is to use the same loss function at test time that was used during training. If you trained against cross-entropy, you report test cross-entropy. Mixing them is technically legal but makes comparison across runs awkward and is rare in practice.

A practical detail: at evaluation time, batches are run with model updates disabled and stochastic behavior (dropout, data augmentation, batch normalization in train mode) turned off. In PyTorch this is model.eval() plus torch.no_grad(); in TensorFlow/Keras the equivalent is training=False. Forgetting this is a classic source of test losses that look slightly worse than they should.

Mean squared error

MSE is the default for regression. It is smooth, differentiable everywhere, and integrates nicely with gradient-based optimization. The squaring makes it sensitive to large errors, so a model that is mostly right but has a handful of catastrophic predictions will show a high MSE. Sometimes that sensitivity is what you want, when rare large mistakes are unusually costly. Sometimes it is not, in which case MAE is a better fit.

Mean absolute error

MAE measures the average absolute deviation between predictions and targets. It is robust to outliers because it does not square errors. The cost is that MAE is not differentiable at zero, which can complicate optimization, though subgradients work fine in practice. An MAE of 4.2 on a temperature regression means the model is off by 4.2 degrees on average.

Categorical cross-entropy

Cross-entropy is the standard loss for classification. It compares the model's predicted probability distribution over classes to a one-hot or soft target distribution. The loss is high when the model puts low probability on the right class, and the penalty grows sharply as the predicted probability approaches zero. A model that is confidently wrong gets punished more than one that is hedging. You can technically use MSE for classification, but it tends to give weak gradients when the model is far from the truth, which is why cross-entropy is preferred.

How does test loss differ from training and validation loss?

Reading the three numbers together tells you more than any one in isolation.

Scroll sideways for more →

A few patterns come up over and over:

• Training loss low, validation loss low, test loss low. The model fit and generalized.
• Training loss very low, validation loss high. Classic overfitting. The model memorized noise in the training set. Test loss will usually be high too. Common fixes are more data, more regularization, smaller model capacity, or earlier stopping.
• Training loss high, validation loss high. Underfitting. The model lacks capacity or did not converge. Fixes: bigger model, longer training, better features, or a less restrictive regularizer.
• Training loss low, validation loss low, but test loss noticeably worse. This usually points to either a distribution shift between validation and test splits or leakage between validation and training data that did not affect the test set.

Google's machine learning crash course recommends three checks when training and test losses diverge: simplify the model by reducing features, increase the regularization rate, and verify that training and test sets are statistically equivalent. ^[4] The third is easy to skip and surprisingly often the real problem.

One quirky case: validation loss below training loss. This sounds impossible but happens routinely with regularization like dropout, which is only active during training. The training-time loss includes the noise injected by dropout; the validation-time loss does not.

Why must the test set be held out?

The whole point of training a model is to use it on data it has not seen yet. Training loss tells you how well the model fits the data you already have, and a model with zero training loss and no other information is essentially a lookup table. What you actually care about is performance in the wild, and the test loss is the closest pre-deployment proxy for that.

If the test set is representative, the test loss is a useful prediction of real-world performance. If it is not, the test loss is misleading in ways that the number alone cannot detect. Thoughtful practitioners pair test loss with sliced metrics, error analysis on held-out examples, and, when possible, a separate validation against fresh data after deployment. For benchmark comparisons in language modeling, vision, and reinforcement learning, test loss is one of the standard reporting metrics, sometimes alongside task scores like accuracy, F1, or BLEU.

The Wikipedia article on training, validation, and test data sets states the rule directly: "the test data set should never be used for validating the training model or fine-tuning hyperparameters." ^[1] Once a test set has informed even a single design decision, it has leaked into the development loop and its loss estimate is no longer unbiased.

What are common pitfalls when measuring test loss?

• Tuning on the test set. If you adjust hyperparameters, architectures, or features based on what improves test loss, the test set has become a validation set and the test loss is no longer unbiased. The fix is the standard three-way split: train, validate, then test exactly once. ^[1]
• Data leakage. Information from the test set sneaking into training, often through preprocessing. Examples: fitting a scaler on the full dataset before splitting, using target statistics computed across train and test, or letting future data appear in the training fold. Leakage produces test losses that look great until the model is deployed. ^[5][8]
• Repeated test set reuse. Running many models against the same test set and reporting the best one is a form of multiple comparison. The reported test loss will be optimistic.
• Mismatched preprocessing. Applying different normalization, tokenization, or feature engineering at test time than at training time. Small differences can dominate the apparent test loss.
• Train/eval mode bugs. Forgetting to switch to evaluation mode leaves dropout, batch norm running statistics, or data augmentation active during test computation. This inflates test loss and is one of the most common bugs in PyTorch and TensorFlow code.
• Non-representative test sets. A model that aces a clean test set can perform badly on messy real inputs.
• Comparing test losses across different loss functions. A cross-entropy of 0.3 and an MSE of 0.3 are not the same thing. Stick to one loss within a comparison group, or convert to a task metric.

What protocols produce a test loss number?

There are two main protocols for producing a test loss number. The simple holdout split, where you set aside a single test partition, is the standard for large datasets and most deep learning work. It is fast and the test loss has well-understood properties as long as the split is random and stratified where appropriate. The downside is variance: a single split can be lucky or unlucky, and on small datasets the resulting estimate is noisy.

K-fold cross-validation cycles through k different train/test splits, computes a test loss on each, and averages. This reduces variance and is the standard approach for tabular machine learning with modest data. The price is k times the compute. When reporting cross-validated test loss, include the standard deviation across folds, not just the mean. ^[6] Nested cross-validation, with an inner loop for hyperparameter tuning and an outer loop for evaluation, is more rigorous when both selection and evaluation must happen on limited data, but is rarely used in deep learning at scale.

How does test loss relate to bias and variance?

The gap between training loss and test loss is closely tied to the bias-variance tradeoff. A model with high bias underfits and shows high training loss and high test loss, with the two roughly equal. A model with high variance overfits and shows low training loss with a much higher test loss. Tuning regularization, model capacity, and training duration is largely about pushing this tradeoff toward whichever point minimizes test loss. ^[7]

In modern deep learning, particularly with overparameterized neural networks, the classical bias-variance picture becomes less tidy. The double descent phenomenon (see below) shows that test loss can decrease again as model size grows past the interpolation point. The intuition still applies, but the simple "bigger model means worse generalization" heuristic does not hold reliably. ^[9]

How does test loss behave under scaling laws?

In large-scale deep learning, test loss is not just a pass/fail check at the end; it is the quantity that scaling laws predict as a smooth function of resources. In "Scaling Laws for Neural Language Models," Kaplan, McCandlish, Henighan, Brown, Chess, Child, Gray, Radford, Wu, and Amodei (2020) found that for Transformer language models, "the loss scales as a power-law with model size, dataset size, and the amount of compute used for training, with some trends spanning more than seven orders of magnitude." ^[10]

Fitting on the WebText2 set, they reported the held-out cross-entropy loss L as separate power laws in the non-embedding parameter count N, the dataset size D in tokens, and the compute budget C, with approximate exponents as follows. The authors note these specific constants depend on tokenization and have no fundamental meaning, but the power-law form is the robust finding.

Scroll sideways for more →

A notable practical takeaway: "Larger models are significantly more sample-efficient, such that optimally compute-efficient training involves training very large models on a relatively modest amount of data." ^[10] These scaling curves are why test loss (often reported as cross-entropy or bits-per-character/token) is the headline metric used to forecast how much compute and data a target loss will require, a methodology later refined by the Chinchilla compute-optimal analysis from DeepMind.

What is double descent in the test loss curve?

The shape of the test loss curve as model size grows is itself a research subject. Classical statistics expects a U-shaped curve: test loss falls, then rises as the model overfits. Modern overparameterized networks instead exhibit "double descent." In "Deep Double Descent: Where Bigger Models and More Data Hurt," Nakkiran, Kaplun, Bansal, Yang, Barak, and Sutskever (2019) showed that "as we increase model size, performance first gets worse and then gets better." ^[9] The test loss peaks near the interpolation threshold, the point where the model is just large enough to fit the training set exactly, and then declines again as capacity grows further.

The paper defines a complexity measure it calls the "effective model complexity" and demonstrates the same non-monotone curve across CNNs, ResNets, and Transformers, across model size, training time, and dataset size. The effect is most visible with label noise. The authors also report the counterintuitive result that "increasing (even quadrupling) the number of train samples actually hurts test performance" in a regime near the threshold. ^[9] Double descent is the clearest example of why, in deep learning, a rising test loss does not always mean you have gone too far: pushing past the interpolation point can lower test loss again.

Explain Like I'm 5 (ELI5)

Imagine studying for a math test. You practice with a workbook (the training set). Halfway through, your tutor gives you a quiz with different problems to check how you are doing (the validation set). On test day, your teacher hands you brand new problems you have never seen (the test set). How well you do on those is your test loss.

A low test loss means you actually understood the math. A high test loss means you may have memorized the practice problems without learning the underlying idea. The reason your teacher does not let you peek at the test ahead of time is the same reason machine learning engineers keep the test set sealed off until the end: peeking would let you tune your answers to the specific questions and make the score look better than your real understanding deserves.

References

1. Wikipedia, Training, validation, and test data sets ↩
2. Wikipedia, Generalization error ↩
3. Wikipedia, Empirical risk minimization ↩
4. Google for Developers, Overfitting: Interpreting loss curves ↩
5. IBM, What is Data Leakage in Machine Learning? ↩
6. scikit-learn documentation, Cross-validation: evaluating estimator performance ↩
7. Goodfellow, Bengio, Courville, Deep Learning, Chapter 5 (Machine Learning Basics) and Chapter 11 (Practical Methodology) ↩
8. scikit-learn documentation, Common pitfalls and recommended practices ↩
9. Nakkiran, Kaplun, Bansal, Yang, Barak, Sutskever (2019), Deep Double Descent: Where Bigger Models and More Data Hurt ↩
10. Kaplan, McCandlish, Henighan, Brown, Chess, Child, Gray, Radford, Wu, Amodei (2020), Scaling Laws for Neural Language Models ↩
11. Machine Learning Mastery, How to use Learning Curves to Diagnose Machine Learning Model Performance
12. ML Glossary, Loss Functions