Training loss

See also: Loss function, Validation loss, Learning curve

In machine learning, training loss is the value of the loss function computed on the training data during model training, and it is the exact quantity that the optimization algorithm minimizes at each step. Formally it is the empirical risk: the average loss over the training examples, written as the average over the training set of the per-example loss between the model's prediction and the target.^[12][14] Google's Machine Learning Glossary defines the underlying loss simply as "a measure of how far a model's prediction is from its label," with a loss function being "a mathematical function that calculates the loss on a batch of examples."^[13] Tracking training loss over iterations or epochs is the primary signal practitioners use to judge whether training is working.^[4]

What is training loss?

Training loss is the numerical value produced when a model's current predictions on the training set are scored by its loss function and averaged. If the training set has N examples, the model parameters are theta, the per-example loss is L, the model's prediction for input x_i is f(x_i; theta), and the target is y_i, then the training loss is (1/N) times the sum over i of L(f(x_i; theta), y_i). This average-loss-on-the-training-set quantity is what statisticians call the empirical risk, and minimizing it is the principle known as empirical risk minimization (ERM).^[12][14] In the formal framing used by the Dive into Deep Learning textbook, the training error is "a statistic calculated on the training dataset," in contrast to the generalization error, which is "an expectation taken with respect to the underlying distribution."^[14]

Optimization algorithms such as gradient descent push this number downward: at each step the optimizer computes the gradient of the training loss with respect to the parameters and nudges the parameters in the direction that reduces it. The same loss function generates a different training loss at every step, and that sequence of numbers is what gets logged, plotted, and analyzed.

A well-behaved training loss curve generally starts high, drops sharply in the first few thousand steps, then settles into a long, gentler decline as the model approaches a minimum of its training objective. Google's glossary describes the artifact used to watch this, the loss curve, as "a plot of loss as a function of the number of training iterations."^[13]

The distinction between loss function and training loss matters. "Loss function" refers to a mathematical formula (such as squared error or cross-entropy), while training loss is the numerical value that formula produces when fed the model's current predictions on the training set. The loss function is a mathematical object; training loss is one of many values that object takes on as the model evolves.

What are the most common loss functions used during training?

The choice of loss function depends on the task. Two are by far the most common starting points. For regression, mean squared error (MSE) averages the squared differences between predictions and targets, computed as (1/N) times the sum of (y_i minus y-hat_i) squared.^[15] For classification, cross-entropy (the negative log-likelihood of the correct class) measures how far the predicted probability distribution is from the true label; for the multi-class case it is the negative average over examples and classes of y_{i,k} times log(p_{i,k}).^[15] A useful unifying fact from Goodfellow et al. is that any loss consisting of a negative log-likelihood is a cross-entropy between the empirical distribution defined by the training set and the model's distribution, and MSE itself is the cross-entropy between the empirical distribution and a Gaussian model.^[4]

The table below summarizes the most common training objectives and where each is used.

Scroll sideways for more →

For LLM pretraining, the training loss is almost always the average per-token negative log-likelihood across the batch, which is the same quantity used to compute perplexity. The relationship is simply perplexity = exp(per-token NLL), so a training loss of 2.0 corresponds to a perplexity of about 7.39.^[12]

How is training loss different from validation loss and test loss?

A machine learning workflow normally tracks loss on three different data partitions, each serving a separate purpose.

Scroll sideways for more →

Training loss is the easiest to drive toward zero because the model has direct access to those examples through gradient updates. Validation loss is the diagnostic that reveals whether the model is learning generalizable patterns or just memorizing the training set. Test loss exists to keep the validation set honest: if the validation loss is repeatedly used to select hyperparameters, it begins to leak information into the model and stops being a fair estimate of true generalization.^[12]

The key reason all three exist is that a low training loss does not by itself guarantee a good model. As the Dive into Deep Learning textbook puts it, "low training error alone is not enough to certify low generalization error," so practitioners "must rely more heavily on our holdout data to certify generalization after the fact."^[14] Google's glossary names the failure mode this guards against, overfitting, as "creating a model that matches the training data so closely that the model fails to make correct predictions on new data."^[13]

A common mistake is conflating training loss with the loss function itself. The loss function is a mathematical object. Training loss is one of many values that function takes on as the model evolves.

Per-batch loss vs per-epoch loss

Training loss is reported at two granularities, and confusing them causes a lot of frustration when reading curves.

Per-batch loss is the loss computed on a single mini-batch right after the gradient update. It is logged every step (sometimes every few steps to reduce I/O) and is inherently noisy, because each batch contains a different sample of training examples. A per-batch curve that bounces up and down by 10 to 20 percent from step to step is not necessarily a problem; it is mostly sampling noise.^[10]

Per-epoch loss is the average loss over an entire pass through the training set. It is much smoother because the noise from individual batches gets averaged out. Most published learning curves show per-epoch loss for clarity, but per-batch logs are valuable for catching short-lived issues like a corrupted batch or a sudden divergence event.

When training runs are long (LLM pretraining can last for weeks across tens of thousands of steps), people often plot a moving average of the per-batch loss using a window of 100 to 500 steps. This gives the smoothness of per-epoch loss with the temporal resolution of per-batch logging.

What does a training loss curve tell you?

The shape of the training loss curve carries a lot of diagnostic information. A few common patterns and what they typically mean:

A steady downward curve that gradually flattens is the textbook healthy training run. The model is converging toward a minimum.

A sudden plateau early in training can mean the learning rate is too small, the model is stuck near a saddle point,^[7] or the data is not being shuffled properly so each epoch presents the same easy examples first.^[9]

Loss spikes (the curve jumps up sharply, then comes back down) usually indicate that a batch contained unusual examples or that the learning rate is too aggressive for that point in training. A single spike is not always fatal: many training runs recover. Repeated spikes are a warning sign.

Divergence (loss climbs steadily or jumps to NaN) means something is broken. Common causes include a learning rate that is too high, exploding gradients, numerical overflow in mixed-precision training, or a data preprocessing bug that is feeding garbage into the model.

A training loss that drops to essentially zero very quickly suggests the model has memorized the training set or, more concerning, that there is data leakage.^[9] If the task is genuinely hard and the model is small, near-zero training loss within a few epochs probably means something is wrong with how the data is being constructed.

How do you read train vs validation loss patterns?

Reading training loss in isolation is rarely enough. Plotting it alongside validation loss reveals whether the model is generalizing.^[10]

Scroll sideways for more →

The gap between training and validation loss, called the generalization gap, is itself a metric worth watching. Dive into Deep Learning defines it directly as "the generalization gap (R_emp minus R) between our training and generalization errors," the difference between the empirical risk on the training set and the true risk on the underlying distribution.^[14] A small gap suggests the model would generalize even further with more data or longer training. A large gap suggests the model has reached the limits of what its current capacity can extract from the available training data without overfitting.^[4] Goodfellow et al. frame the entire challenge of machine learning in these two terms: an algorithm performs well when it can make the training error small and also make the gap between training error and test error small, with underfitting corresponding to a training error that is too high and overfitting corresponding to a gap that is too large.^[4]

What else should you log alongside training loss?

Training loss alone is rarely sufficient to debug a training run. Modern experiment trackers log a constellation of related metrics that together paint a much richer picture.

Scroll sideways for more →

Logging gradient norm in particular is one of the most useful early-warning signals. A gradient norm that suddenly doubles a few steps before a loss spike often makes the failure predictable.

Tools for monitoring training loss

Several platforms have become standard for tracking training loss and related metrics across the deep learning ecosystem.

Scroll sideways for more →

TensorBoard remains the most universally supported option. In PyTorch, the typical pattern is to instantiate a SummaryWriter and call writer.add_scalar("Loss/train", loss.item(), global_step) inside the training loop. Hierarchical tag naming ("Loss/train", "Loss/val") groups related curves automatically in the UI. The TensorBoard documentation recommends calling writer.close() at the end of training so all buffered events are flushed to disk.^[8]

Weights and Biases became popular for cloud-hosted runs because it makes it trivial to compare hundreds of experiments side by side, run hyperparameter sweeps, and share results with collaborators via a public URL. MLflow is more common in enterprise settings that already use Databricks or need to integrate with broader MLOps tooling.

Special considerations for LLM training

Large language model pretraining stretches every assumption about loss curves. Runs last days or weeks, models have hundreds of billions of parameters, and even small instabilities can cost tens of thousands of dollars in wasted compute.

Loss spikes

Loss spikes during LLM pretraining are well documented. The PaLM paper (Chowdhery et al., 2022) reported that the training loss spiked roughly 20 times during the run for the 540B parameter model, even though gradient clipping was active.^[1] The team's mitigation was practical rather than theoretical: when a spike occurred, they restarted training from a checkpoint about 100 steps before the spike and skipped the next 200 to 500 batches that the model had been about to consume. Skipping the batches almost always prevented the spike from recurring at the same point, suggesting the spikes were caused by specific combinations of data and model state rather than fundamental optimizer failure.^[1]

The Spike No More paper (Takase et al., 2023)^[3] and the Llama 3 technical report^[11] both document similar phenomena and propose more principled mitigations including improved initialization, layer normalization placement, and adaptive gradient clipping.

Gradient clipping and numerical stability

Gradient clipping (typically clipping the global norm to 1.0) is standard in LLM training to prevent rare large gradients from blowing up the loss. BF16 (bfloat16) has largely replaced FP16 for large-scale training because BF16 has the same exponent range as FP32, eliminating the overflow issues that required loss scaling in FP16.^[11] Mixed-precision training in BF16 produces noticeably smoother loss curves than FP16 at the same model scale.

Loss vs perplexity

LLM training loss is the per-token cross-entropy. Perplexity, the more interpretable quantity, is just exp(per-token loss). A loss of 2.5 corresponds to a perplexity of about 12.2, meaning the model is on average about as uncertain as if it were choosing uniformly among 12 possible next tokens.^[12] Both quantities are reported in practice, with loss preferred during optimization (because gradients flow through it directly) and perplexity preferred when discussing model quality.

The loss landscape

The training loss can be thought of as a surface in the high-dimensional space of model parameters. Each combination of weights produces one loss value, and the optimizer's job is to walk downhill on this surface. Visualizing this landscape is mathematically tricky because deep models have millions to billions of parameters, but Li et al. (NeurIPS 2018) introduced a filter normalization technique that produces meaningful 2D and 3D visualizations of loss landscapes.^[2] Their key empirical finding was that flatter minima tend to generalize better than sharp ones, supporting earlier theoretical work by Hochreiter and Schmidhuber on the relationship between minimum sharpness and generalization.^[6]

The paper also showed that skip connections (as in ResNet architectures) dramatically smooth the loss landscape, which helps explain why very deep networks became trainable once skip connections were introduced.^[2] Without skip connections, the loss landscape of a deep network is full of sharp ridges and chaotic regions; with them, the landscape becomes much closer to the gentle bowl that gradient descent handles well.

This line of work also clarified that the training loss landscape exhibits implicit regularization: stochastic gradient descent with reasonable hyperparameters tends to find flat minima that generalize, even when sharper minima with the same training loss exist.

Common mistakes when working with training loss

A few patterns trip up practitioners often enough to be worth flagging:

Comparing losses across different scales. MSE on raw pixel values produces numbers in the thousands, while cross-entropy on a 50,000-token vocabulary produces numbers around 10. Comparing absolute values across loss functions is meaningless. Compare relative changes within the same loss function.^[4]

Reporting training loss inconsistently. Some pipelines include the regularization term (weight decay, L2 penalty) in the reported training loss, others do not. When comparing runs across codebases, check whether "training loss" includes the regularizer.

Forgetting to call model.eval() when computing validation loss. In PyTorch, this leaves dropout active and uses batch statistics rather than running statistics in batch normalization layers. The result is a noisy, biased validation loss that does not reflect inference behavior.

Not normalizing by batch size in distributed training. When gradients are summed across workers, the effective batch size grows, and the loss should be averaged appropriately. Forgetting this leads to learning rates that are effectively too large or too small.

Mixing per-batch and per-epoch loss in the same plot without saying which is which. The two have very different smoothness, and putting them on the same axes invites misinterpretation.

Reading too much into a single noisy curve. A loss that bounces around within a 10 percent band over 100 steps is normal noise, not evidence of instability. Take a moving average before drawing conclusions.

Crash diagnostics versus convergence diagnostics

For LLM-scale runs, monitoring tooling has split into two functional categories. Crash diagnostics are designed to detect divergence as early as possible: rapidly rising loss, gradient norm spikes, NaN detection, and weight norm growth all qualify. The faster these signals trip an alert, the less compute is wasted on a doomed run. Convergence diagnostics, by contrast, are slower-moving statistics that help judge whether a run that is technically training is also making efficient progress: comparison against scaling-law predictions, downstream evaluation metrics, and per-domain perplexity all fall into this bucket. A well-designed monitoring setup logs both categories and surfaces them in different dashboards so on-call engineers can react to crashes without being overwhelmed by slow trends.

Explain like I'm 5 (ELI5)

Imagine you are practicing free throws in basketball. Each time you shoot, the coach writes down how far the ball was from the center of the hoop. The numbers usually start big (you keep missing badly) and get smaller as you practice. Training loss is the same idea for a computer program that is learning. Every time the program tries to make a guess, the training loss says how wrong the guess was. The program changes itself a little bit to be less wrong next time, and the training loss number gets smaller and smaller.

If the number ever stops going down, that might mean you have learned everything you can from this practice or that something is going wrong, like the gym lights flickering and confusing you. And if the number suddenly jumps up, that probably means a weird ball came along that messed you up. The score by itself does not tell you everything; you also need to try shooting in a different gym (the validation set) to know whether you are really getting better or just memorizing the lines on the floor of one specific court.

References

1. Chowdhery, A., et al. (2022). "PaLM: Scaling Language Modeling with Pathways." https://arxiv.org/abs/2204.02311 ↩
2. Li, H., Xu, Z., Taylor, G., Studer, C., & Goldstein, T. (2018). "Visualizing the Loss Landscape of Neural Nets." *NeurIPS 2018*. https://arxiv.org/abs/1712.09913 ↩
3. Takase, S., Kiyono, S., Kobayashi, S., & Suzuki, J. (2023). "Spike No More: Stabilizing the Pre-training of Large Language Models." https://arxiv.org/abs/2312.16903 ↩
4. Goodfellow, I., Bengio, Y., & Courville, A. (2016). *Deep Learning*, Chapters 5 and 8. MIT Press. https://www.deeplearningbook.org/ ↩
5. Lin, T.-Y., Goyal, P., Girshick, R., He, K., & Dollar, P. (2017). "Focal Loss for Dense Object Detection." *ICCV 2017*. https://arxiv.org/abs/1708.02002 ↩
6. Hochreiter, S., & Schmidhuber, J. (1997). "Flat Minima." *Neural Computation*, 9(1), 1-42. ↩
7. Dauphin, Y., Pascanu, R., Gulcehre, C., Cho, K., Ganguli, S., & Bengio, Y. (2014). "Identifying and attacking the saddle point problem in high-dimensional non-convex optimization." *NeurIPS 2014*. https://arxiv.org/abs/1406.2572 ↩
8. PyTorch Documentation. "torch.utils.tensorboard." https://pytorch.org/docs/stable/tensorboard.html ↩
9. Google Developers Machine Learning Crash Course. "Overfitting: Interpreting loss curves." https://developers.google.com/machine-learning/crash-course/overfitting/interpreting-loss-curves ↩
10. Hugging Face. "Understanding Learning Curves." *LLM Course, Chapter 3.5*. https://huggingface.co/learn/llm-course/en/chapter3/5 ↩
11. Llama Team, AI @ Meta (2024). "The Llama 3 Herd of Models." https://arxiv.org/abs/2407.21783 ↩
12. Murphy, K. P. (2022). *Probabilistic Machine Learning: An Introduction*, MIT Press. https://probml.github.io/pml-book/book1.html ↩
13. Google for Developers. "Machine Learning Glossary." (entries: loss, loss function, loss curve, overfitting). https://developers.google.com/machine-learning/glossary ↩
14. Zhang, A., Lipton, Z. C., Li, M., & Smola, A. J. "Generalization." *Dive into Deep Learning*. https://d2l.ai/chapter_linear-regression/generalization.html ↩
15. Wang, Q., Ma, Y., Zhao, K., & Tian, Y. (2022). "A Comprehensive Survey of Loss Functions in Machine Learning." *Annals of Data Science*. https://arxiv.org/abs/2301.05579 ↩