Cost

See also: Machine learning terms

In machine learning, cost is the scalar number that summarizes how badly a model is performing on a chunk of data. The function that produces it is the cost function, often written J(θ), where θ is the vector of model parameters. A training algorithm changes θ to push J(θ) downward, and that single number is the only thing the optimizer ever sees. Whatever J encodes, the model will learn to do; whatever J ignores, the model will treat as free to wreck.

In casual conversation, ML practitioners use "cost," "loss," and "objective" interchangeably, and the standard textbook (Goodfellow, Bengio, and Courville's Deep Learning) explicitly says the function we want to minimize is called the objective function, criterion, cost, or loss, with no real distinction ^[1]. In the book's exact words: "The function we want to minimize or maximize is called the objective function, or criterion. When we are minimizing it, we may also call it the cost function, loss function, or error function." ^[1] When people do bother to draw a line, the most common convention is the one Sebastian Raschka uses on his FAQ: loss is per example, cost is the aggregate over a dataset, and objective is the broadest umbrella term that may include a regularizer or even maximization ^[2]. This article focuses on the cost view, and on the second meaning that has crept into modern ML: the dollars, watts, and GPU-hours it takes to actually train and serve a model.

what is the cost function?

The canonical machine-learning cost function is the regularized empirical average

$$J(\theta) = \frac{1}{N} \sum_{i=1}^{N} L(f_\theta(x_i), y_i) + \lambda R(\theta)$$

where L is a per-example loss function, $f_\theta$ is the model, $(x_i, y_i)$ is the i-th input-label pair from a dataset of size N, $R(\theta)$ is a regularization penalty, and $\lambda \ge 0$ controls how much the regularizer matters. When $\lambda = 0$ the cost reduces to the empirical risk and the setup is the textbook empirical risk minimization (ERM) problem of Vapnik ^[4]. When $\lambda > 0$, the cost is a Pareto trade-off between fitting the training data and keeping $\theta$ small or simple.

The cost is what the optimizer minimizes; everything else (accuracy on validation, calibration, latency, fairness, dollars per inference) is a secondary metric the team tracks but the gradient never sees.

how do cost, loss, risk, and objective differ?

The vocabulary is overloaded across textbooks, papers, and frameworks. The table below shows how careful authors distinguish the terms.

Scroll sideways for more →

The loss-vs-cost split is a convention, not a law. Andrew Ng's Coursera ML course popularized writing L for the per-example version and J for the aggregated version, and the Baeldung CS notes follow the same split ^[3]. PyTorch and TensorFlow both call them "losses" and apply a reduction='mean' argument to do the aggregation; in those frameworks there is no separate "cost" object ^[17].

The risk-vs-cost split is the load-bearing one. Cost is sample-based and computable; risk is population-based and only ever estimated. The whole point of generalization theory is that minimizing the cost (what we can compute) does not automatically minimize the risk (what we care about). Léon Bottou's writings on stochastic gradient descent are explicit about this: SGD optimizes the empirical risk while we hope it generalizes to the expected risk, and the gap between them is the entire field of statistical learning theory ^[5].

common cost functions

The per-example loss L drops into the same averaging shell. The choice of L is what defines the task. The table below collects the formulas you will encounter in nearly any production codebase.

Scroll sideways for more →

A few of these merit a longer look.

Focal loss, introduced by Lin et al. at ICCV 2017 with their RetinaNet detector, multiplies the standard cross-entropy by $(1 - p_t)^\gamma$ so that easy, well-classified examples contribute almost nothing to the gradient ^[8]. The motivation came from one-stage object detectors that have to score 10K to 100K candidate boxes per image, almost all of them background ^[8]. Without down-weighting the easy negatives, they swamp the loss and the detector never learns to find the rare positives.

The DDPM noise-prediction loss of Ho, Jain, and Abbeel (2020) is just an MSE between the actual Gaussian noise sampled in the forward process and the noise the network predicts ^[10]. It is the simplest possible objective in the deep generative literature, and it is the foundation under Stable Diffusion, Imagen, and most large-scale image and video diffusion systems ^[10]. The original derivation goes through a variational lower bound, but the simplified loss is what is actually trained ^[10].

The PPO clipped surrogate of Schulman et al. (2017) is the inner optimization step in reinforcement learning from human feedback (RLHF) for LLMs, among many other RL applications ^[9]. The clip prevents the importance-sampling ratio $r_t = \pi_\theta(a \mid s) / \pi_{\text{old}}(a \mid s)$ from straying too far from 1, which keeps each policy update small and stable ^[9].

DPO, the direct-preference-optimization loss of Rafailov et al. (NeurIPS 2023), is a binary cross-entropy on the difference of log-ratios between a policy and a fixed reference policy ^[11]. The closed-form derivation collapses the entire RLHF pipeline (reward model, PPO loop, KL penalty) into a single supervised loss on preference pairs ^[11]. The β temperature controls how strongly the policy is pulled toward the preference signal versus held near the reference.

regularized cost: ridge, lasso, weight decay

When R(θ) is added to the average loss, the cost punishes both poor fit and large or complex parameters. The two most common choices are L2 and L1 penalties, with names that vary by community.

Scroll sideways for more →

In linear regression, the ridge cost is $J(\theta) = \lVert y - X\theta \rVert^2 + \lambda \lVert \theta \rVert^2$, with closed-form solution $\theta = (X^\top X + \lambda I)^{-1} X^\top y$. The lasso version replaces $\lVert \theta \rVert^2$ with $\lVert \theta \rVert_1$ and gives the celebrated sparse solutions of Tibshirani (1996) ^[7]. For deep nets, weight decay (L2 with a small λ, often around 1e-4) is the default regularizer, applied either through the loss or, in modern AdamW, decoupled from the gradient step entirely.

cost-sensitive learning

In many real applications the cost of getting an answer wrong is not symmetric. Missing a malignant tumor (a false negative) is much worse than calling a benign mole malignant (a false positive). Cost-sensitive learning modifies the cost function so that different mistakes are weighted differently.

The simplest version replaces the uniform 1/N average with class-weighted or sample-weighted averaging:

$$J(\theta) = \frac{1}{N} \sum_i w_i L(f_\theta(x_i), y_i)$$

where w_i depends on the true class y_i (class weights) or on the specific example (sample weights). For binary classification with a 2x2 cost matrix C, the per-example loss becomes a weighted sum over the four (true, predicted) cells, and the optimal classifier shifts the decision threshold away from 0.5 toward whichever class is cheaper to predict.

The medical-screening literature is full of this: Rao et al. (2024) review cost-sensitive learning across imbalanced medical data and note that the choice of cost ratio can change AUROC, precision-recall, and the deployed sensitivity of the screen by tens of percentage points ^[16]. scikit-learn implements this through class_weight='balanced' on most estimators, and PyTorch through the weight argument to nn.CrossEntropyLoss ^[17].

what makes a good cost function?

A cost function only earns its place if it has the right mathematical properties for the optimizer it will be paired with.

Scroll sideways for more →

The non-convex part is the painful one. Almost no neural network cost is convex, which is why theoretical guarantees for convex optimization only loosely transfer to deep learning. In practice the loss surface is full of saddle points and local minima; SGD with momentum and Adam happen to find good ones often enough that we ship the model anyway.

connection to optimization

The cost function and the optimizer are paired. You design J(θ) so that an algorithm can minimize it; the optimizer takes gradients (or finite differences, or surrogate models) of J and walks θ downhill. Three concepts come up over and over.

The cost surface is the graph of J(θ) over the parameter space. For a deep net it lives in millions or billions of dimensions and is impossible to visualize directly; the popular "loss landscape" plots project it to two dimensions by sweeping along chosen directions in θ-space. Local minima are points where $\nabla J = 0$ and the Hessian is positive semi-definite; saddle points are points where $\nabla J = 0$ but the Hessian has both positive and negative eigenvalues. Goodfellow et al. argue that in high dimensions saddle points dominate, and SGD escapes them by adding the right kind of noise ^[1].

Gradient descent updates $\theta_{t+1} = \theta_t - \eta \nabla J(\theta_t)$. For a full-batch gradient over the entire dataset, each step is exact but expensive. Stochastic gradient descent estimates ∇J from a single example or a mini-batch, which is what makes large-scale ML feasible ^[5]. Bottou's chapter in Neural Networks: Tricks of the Trade (2012) and the survey "Optimization Methods for Large-Scale Machine Learning" (Bottou, Curtis, Nocedal 2018) lay out the convergence theory ^[5][6]. Adam, AdamW, RMSProp, and the rest of the modern menagerie are variants that adapt the step size per parameter using running estimates of the first and second moments of the gradient.

The practical point is that the gradient is computed by autograd from the cost; if the cost is wrong, the gradient is wrong, and the optimizer obediently walks the wrong way.

how much does it cost to train and serve an AI model?

Machine-learning practitioners increasingly use "cost" in a second sense that has nothing to do with J(θ): the actual money and electricity spent on training and serving a model. As models scaled past GPT-3, this kind of cost stopped being a footnote and became a strategic constraint.

Scroll sideways for more →

The Strubell, Ganesh, and McCallum 2019 paper at ACL was the first widely cited attempt to put numbers on the carbon side of this ^[12]. They estimated that training a single large NLP model with neural-architecture search produced roughly the same lifetime CO2 emissions as five average American cars: an estimated 626,155 pounds of CO2 equivalent for the full search-and-tuning pipeline, which itself required training 4,789 models over a six-month period ^[12][20]. The paper pushed the field to start reporting energy and dollar costs alongside accuracy, and it has been hugely influential in the green-AI conversation and in the more recent push for energy-efficient training.

For frontier LLMs the dollar numbers are public, if approximate. Sam Altman pegged the training cost of GPT-4 at "more than $100 million," and Stanford's 2025 AI Index along with Epoch AI estimate the compute portion alone at around $78 million ^[13][14]. Llama 3.1 405B is estimated by the same Stanford report at roughly $170 million in compute, with other analysts citing figures as low as $60 million depending on whether amortized hardware capex is included; Meta trained it on over 16,000 H100 GPUs for roughly 30.8 million GPU-hours ^[13][19]. Google's Gemini Ultra came in around $191 million by the same methodology ^[13]. These numbers are rounded, contested, and almost always exclude staff, R&D, and failed runs, but they are the right order of magnitude.

Inference cost has its own pricing world. Anthropic's Claude Opus 4.8, the current flagship released in May 2026, is priced at $5 per million input tokens and $25 per million output tokens, with a 1M-token context window on by default ^[18]. The prior Claude Opus 4.7 carries the same rates; Sonnet 4.6 sits at $3 / $15, and Haiku 4.5 at $1 / $5 ^[15]. Batch processing across all of these is 50% cheaper, and prompt caching can drop the effective input cost by up to 90% ^[15]. Inference cost matters because it is paid every single time a user makes a query, while training cost is paid once.

Annotation cost is the unglamorous one. Hiring humans to label millions of images for ImageNet, to write demonstration responses for InstructGPT, or to rank pairwise preferences for RLHF is often the dominant line item for tasks where data is the bottleneck rather than compute. The OpenAI InstructGPT paper noted that the human-feedback dataset was small but expensive, and the entire DPO line of work is partly motivated by squeezing more supervision out of the same preference labels ^[11].

common mistakes

Writing down J is the most consequential step in a training run, and the most common failures are conceptual rather than numerical.

• Confusing cost with risk. The training cost is what we can compute on a sample; the true risk is what we actually care about on the population. They are not the same number, and the gap between them is the whole point of generalization theory and the regularizer term.
• Confusing cost with the evaluation metric. Training MSE while reporting classification accuracy means the model is optimal for MSE, not for accuracy. If the metric is non-differentiable (F1, BLEU, NDCG), use a smooth surrogate for training and report the metric on validation.
• Reduction bugs. PyTorch's reduction='sum' versus reduction='mean' changes the effective learning rate by a factor of batch size, and ignoring padding tokens in a language model makes the per-token mean differ from the per-batch mean. The cost stays correct only if these reductions match what the optimizer expects.
• Numerical instability. Cross-entropy contains log(p) and softmax contains exp(z). Log of zero blows up; exp of large values overflows. Standard fixes include the log-sum-exp trick, fused softmax-cross-entropy kernels (PyTorch's cross_entropy on logits, TensorFlow's softmax_cross_entropy_with_logits), and clamping probabilities into (ε, 1 - ε).
• Scale mismatch between data fit and regularization. If the per-example loss is in the thousands and λR(θ) is order 1, the regularizer does nothing. Always check that the two terms are within a few orders of magnitude of each other early in training.
• Reporting training loss as if it were generalization performance. Training cost going to zero means the model has memorized the training set, a symptom of overfitting, not that it has learned a useful function. The validation cost is the relevant number.
• Treating the deployed cost as the same as the training cost. Inference cost (latency, dollars per token, energy per query) is paid forever; training cost is paid once. Optimizing only one of them ignores the other.

implementations

In practice you almost never write the cost function from scratch. The major ML frameworks ship dozens of pre-built losses, all of them invoked the same way: instantiate, then call on (predictions, targets) to get a scalar.

Scroll sideways for more →

PyTorch's convention is worth a closer look because it shows up in nearly every research paper. nn.CrossEntropyLoss takes raw logits (not probabilities) and an integer class label, computes a numerically stable log-softmax internally, and reduces with mean by default ^[17]. The conventional variable name for the loss object is criterion, which goes back to the statistics term. nn.BCEWithLogitsLoss does the same trick for binary classification: it fuses sigmoid and binary cross-entropy so that the gradient stays stable for very confident predictions.

explain like I'm 5

Imagine you are teaching a robot to throw beanbags into a bucket. After every throw the robot gets a number that says how badly it missed. Big miss, big number. Hit the bucket, small number. That number is the cost. The robot's whole life is figuring out how to wiggle its arm so the next number is smaller than the last one.

The rule that turns a throw into a number is the cost function. Different rules give different robots. If the rule is "how far the bag landed from the bucket," the robot learns to aim for the middle of the bucket. If the rule is "did the bag land in the bucket, yes or no," the robot might learn to just barely tip it over the rim. Pick the wrong rule and you get the wrong robot, even if the wrong robot is very good at being wrong.

The other meaning of cost is even simpler: training a really big robot uses a lot of electricity, costs a lot of money, and sometimes burns through millions of dollars. That is also called cost.

related concepts

The broader objective function article covers the umbrella mathematical concept and includes detailed treatment of convexity, smoothness, and constrained optimization. The loss function article covers the per-example formulas with code examples for individual losses. The empirical risk minimization article gives the formal statistical framework under which minimizing the cost approximates minimizing the true risk, and the training loss article describes the specific case of cost computed on the training set during the inner optimization loop.

references

1. Goodfellow, I., Bengio, Y., and Courville, A. (2016). *Deep Learning*. MIT Press. Chapter 4.3 introduces the terms objective function, criterion, cost, and loss as essentially synonymous. ↩
2. Raschka, S. "What is the difference between a cost function and a loss function in machine learning?" sebastianraschka.com FAQ. ↩
3. Baeldung. "Difference Between the Cost, Loss, and the Objective Function." baeldung.com/cs/cost-vs-loss-vs-objective-function. ↩
4. Vapnik, V. (1998). *Statistical Learning Theory*. Wiley. Empirical risk minimization framework. ↩
5. Bottou, L. (2012). "Stochastic Gradient Descent Tricks." In *Neural Networks: Tricks of the Trade*, Springer. ↩
6. Bottou, L., Curtis, F. E., and Nocedal, J. (2018). "Optimization Methods for Large-Scale Machine Learning." *SIAM Review*, 60(2), 223-311. arXiv:1606.04838. ↩
7. Tibshirani, R. (1996). "Regression Shrinkage and Selection via the Lasso." *Journal of the Royal Statistical Society B*. ↩
8. Lin, T.-Y., Goyal, P., Girshick, R., He, K., and Dollár, P. (2017). "Focal Loss for Dense Object Detection." ICCV. arXiv:1708.02002. ↩
9. Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). "Proximal Policy Optimization Algorithms." arXiv:1707.06347. ↩
10. Ho, J., Jain, A., and Abbeel, P. (2020). "Denoising Diffusion Probabilistic Models." NeurIPS. arXiv:2006.11239. ↩
11. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D., and Finn, C. (2023). "Direct Preference Optimization: Your Language Model is Secretly a Reward Model." NeurIPS. arXiv:2305.18290. ↩
12. Strubell, E., Ganesh, A., and McCallum, A. (2019). "Energy and Policy Considerations for Deep Learning in NLP." ACL. arXiv:1906.02243. ↩
13. Stanford HAI (2025). *Artificial Intelligence Index Report 2025*. Chapter on training cost trends (GPT-4 ~$78M, Llama 3.1 405B ~$170M, Gemini Ultra ~$191M in compute). ↩
14. Epoch AI (2024). "Trends in Training Cost of Frontier AI Systems." epochai.org. ↩
15. Anthropic (2026). "Claude API Pricing." platform.claude.com/docs/en/about-claude/pricing. ↩
16. Rao, A. et al. (2024). "Cost-sensitive learning for imbalanced medical data: a review." *Artificial Intelligence Review*, Springer. ↩
17. PyTorch documentation. `torch.nn.CrossEntropyLoss`, `torch.nn.MSELoss`. pytorch.org/docs. ↩
18. Anthropic (2026). "Introducing Claude Opus 4.8." anthropic.com/news/claude-opus-4-8 (general availability May 28, 2026; $5 / $25 per million tokens; 1M-token context window by default). ↩
19. Grattafiori, A. et al. (2024). "The Llama 3 Herd of Models." arXiv:2407.21783. Llama 3.1 405B pre-trained on up to 16,000 H100 GPUs; the model card reports roughly 30.8 million H100 GPU-hours for the 405B model. ↩
20. Hao, K. (2019). "Training a single AI model can emit as much carbon as five cars in their lifetimes." MIT Technology Review. ↩