Scoring

See also: Machine learning terms

Scoring is the process of applying a trained machine learning model to data so it produces an output number, either a prediction (probability, class score, or ranking value) emitted at inference time, or a metric value that measures how good the model's predictions are. The word covers two related ideas because the mechanic is identical: the model emits a number, and that number is read as a decision input (in inference) or a quality measurement (in evaluation). In recommendation systems specifically, scoring also names a distinct pipeline stage that assigns a relevance value to each candidate item so the items can be ranked, sitting between candidate generation and re-ranking. ^[4]

Overview

In machine learning, scoring has two closely related meanings that often get blurred in practice. The first is evaluation: applying a metric to a model's predictions on labeled data to measure how well the model is performing. The second is inference: running a trained model on new data points to produce predictions, probabilities, or ranking scores that downstream systems can use. The same word covers both because the underlying mechanic is the same. The model emits a number, and that number gets interpreted as either a quality measurement (in evaluation) or a decision input (in production).

The inference sense of scoring is the older one. In credit risk, fraud detection, and direct marketing, "scoring" has meant running a fitted model against a customer record to produce a number since at least the 1980s. The evaluation sense became dominant later, popularized by tools like scikit-learn that exposed a scoring parameter for selecting models against a chosen metric. Both senses share a convention: higher scores mean better outcomes for the receiver, whether that is a more accurate model or a higher predicted likelihood of the event of interest. ^[1]

What happens during scoring at inference time?

When a trained classifier sees a new input, it usually produces more than a hard class label. Most modern classifiers expose a continuous score for each class. In scikit-learn, the predict_proba method returns an array with one column per class, where each column holds the model's estimated probability that the observation belongs to that class. The companion method decision_function returns the raw, uncalibrated score, which for many linear models corresponds to the signed distance from the decision boundary and for neural networks corresponds to the pre-softmax logits.

The relationship between these scores and probabilities depends on the model family. A logistic regression maps its linear score through a sigmoid (or softmax for multiclass) and the output is genuinely a probability under the model's assumptions. A random forest returns the proportion of trees voting for each class. A support vector machine has no native probability and only fits one if you ask for it (sklearn uses Platt scaling for that). Tree ensembles like XGBoost and gradient-boosted forests output a raw margin that becomes a probability after a sigmoid or softmax step. Even when a model exposes a predict_proba method, the numbers it returns are not guaranteed to behave like calibrated probabilities: scikit-learn's calibration documentation notes that GaussianNB "tends to push probabilities to 0 or 1" and that a random forest classifier shows "a characteristic sigmoid shape, indicating that the classifier could trust its 'intuition' more," so for many estimators the raw output is better treated as a relative score than as a literal probability. ^[2]

That distinction matters because scores are not automatically calibrated. The scikit-learn calibration guide states it directly: "Well calibrated classifiers are probabilistic classifiers for which the output of the predict_proba method can be directly interpreted as a confidence level. For instance, a well calibrated (binary) classifier should classify the samples such that among the samples to which it gave a predict_proba value close to, say, 0.8, approximately 80% actually belong to the positive class." ^[2] Many high-performing models are systematically over- or under-confident. Probability calibration techniques such as Platt scaling and isotonic regression are post-processing steps that fit a one-dimensional function from raw scores to calibrated probabilities, typically on a held-out set. Calibration is measured with strictly proper scoring rules: the Brier score (mean squared error between predicted probability and outcome) and log loss (cross-entropy), both of which jointly capture calibration and discrimination. ^[8]

What is the difference between batch scoring and online scoring?

In production systems, scoring is usually described as either batch or online. Batch scoring runs the model on a large set of records on a schedule, perhaps overnight, and writes predictions to a database for downstream use. A credit card issuer might re-score every active account weekly. A retailer might generate next-day product recommendations for every customer once per day. Batch jobs trade latency for throughput and tend to be cheaper per prediction because the infrastructure can be sized for steady utilization. Azure Machine Learning, for example, exposes dedicated batch endpoints for exactly this pattern, deploying a model "for scoring in batch" over large data sets asynchronously rather than per request. ^[12]

Online scoring (also called real-time inference or dynamic inference) responds to individual requests as they arrive, typically through a REST API or RPC endpoint. The scoring path needs to be fast, often under 100 milliseconds, and the system has to handle traffic spikes. Online scoring is the default for use cases where the prediction depends on signals that change in real time: fraud detection during checkout, ranking the next video to autoplay, or routing a support ticket. Some platforms combine both modes in a lambda-style architecture where a batch layer scores everything regularly and a speed layer handles fresh requests in real time.

How does scoring work in recommender systems?

Recommender systems are where the inference sense of scoring is most visible, because production pipelines are almost always built as a two-stage funnel. Candidate generation comes first. Given a user and some context, the system retrieves a few hundred or a few thousand items from a catalog that may contain millions or billions of entries. Candidate generators have to be cheap. They typically rely on approximate nearest-neighbor lookups against embeddings, collaborative filtering, or simple heuristics like "recently popular in your region." ^[5]

Then comes the scoring stage, often just called ranking. A second, heavier model takes the candidate set and assigns each item a relevance score for the specific user in the specific context. Because the candidate set is small, the ranker can afford to consume hundreds of features per item: user history, item metadata, time of day, device, recent session behavior, and cross features built from all of these. Google's recommendation systems course gives the standard reason for this split. Scores from different candidate generators "might not be comparable," and "with a smaller pool of candidates, the system can afford to use more features and a more complex model that may better capture context." ^[4]

The canonical reference for this architecture is the 2016 paper "Deep Neural Networks for YouTube Recommendations" by Paul Covington, Jay Adams, and Emre Sargin, published in the Proceedings of the 10th ACM Conference on Recommender Systems (pages 191-198). ^[6] The paper describes candidate generation as an extreme multiclass classification problem solved with a softmax over millions of videos, followed by a separate deep ranking model. YouTube's ranker famously optimizes expected watch time rather than click probability, a choice that changes which videos win and which lose. Google explicitly warns that the choice of scoring objective changes the system: "If the scoring function optimizes for clicks, the systems may recommend click-bait videos," while "if the scoring function optimizes for watch time, the system might recommend very long videos, which might lead to a poor user experience." ^[4] Combinations are usually needed to balance engagement against diversity.

How are ranking models trained? Learning to rank

The machine learning literature on how to train rankers is grouped into three families based on the granularity of the training signal. Pointwise approaches treat ranking as a regression or classification problem, predicting a relevance score for each item independently of the others. Pairwise approaches train on pairs of items and try to predict which item in the pair is more relevant; RankNet, LambdaRank, and LambdaMART are the well-known pairwise algorithms. Listwise approaches optimize a loss over the full ranked list, directly targeting ranking metrics like NDCG. In practice, listwise methods tend to outperform pairwise, which tends to outperform pointwise, but listwise training is more expensive and harder to scale. ^[7]

Online platforms commonly mix these. A pointwise click-probability model can serve as a strong baseline ranker, with a pairwise or listwise model layered on top for fine-grained sorting. Position bias is another concern at scoring time. Items at the top of a feed naturally attract more clicks regardless of their true relevance, so production rankers either model the position explicitly as a feature, apply inverse-propensity weighting, or score every candidate as if it were in the top slot.

Credit scoring: the original scoring application

Credit scoring is the original "scoring" application and is still the largest by economic impact. The Fair Isaac Corporation (now FICO) was founded in 1956 by engineer William R. "Bill" Fair and mathematician Earl Judson Isaac, and introduced the modern general-purpose FICO Score at Equifax in 1989. ^[9][11] FICO scores run from 300 to 850, a three-digit range that was chosen partly because storage was expensive at the credit bureaus when the format was designed and a three-digit field hit the right balance of precision and compactness.

FICO models are built on what the company calls Scorecard Module technology, which produces interpretable scorecards composed of binned features and additive weights. The resulting score is essentially a calibrated log-odds, mapped to the familiar 300 to 850 range. FICO has used machine learning in its development pipeline for decades, primarily to identify candidate variables and validate features, but the final scoring model that gets shipped is still an interpretable scorecard rather than a neural network or gradient boosted tree. In published research, FICO has compared scorecards against neural networks and gradient boosted trees on identical credit bureau data and found the accuracy gap to be under two percent, which the company argues does not justify giving up interpretability and the ability to generate adverse-action reasons required by the U.S. Equal Credit Opportunity Act. ^[10]

VantageScore, the competing credit score developed jointly by the three major U.S. credit bureaus (Equifax, Experian, and TransUnion), was introduced in March 2006. ^[11] Its first two model generations (VantageScore 1.0 in 2006 and 2.0 in 2010) used a 501 to 990 scale with letter grades, not the FICO range. It was VantageScore 3.0, released in 2013, that adopted the same 300 to 850 range as modern FICO scores to simplify implementation and consumer understanding, and later versions have shifted more aggressively toward machine learning. ^[11] Credit scoring is a useful example of why "score" is the right word: the output is a single number per applicant, intended to be ranked and thresholded by lenders, and the model behind it has to be defensible to regulators on both fairness and adverse-action grounds.

What is the scoring parameter in scikit-learn?

Scikit-learn standardized a particular interpretation of "scoring" through its scoring parameter, which appears in cross_val_score, cross_validate, GridSearchCV, RandomizedSearchCV, and related model-selection tools. The argument tells the tool which metric to optimize when comparing candidate models. It accepts three forms: None (use the estimator's default), a string naming a predefined metric, or a callable scorer. ^[1]

The convention is that higher return values are always better. The documentation states that "all scorer objects follow the convention that higher return values are better than lower return values." ^[1] To handle error metrics that are naturally minimized, scikit-learn provides negated versions: metrics that measure the distance between the model and the data, like mean_squared_error, are exposed as 'neg_mean_squared_error', which returns the negated value of the metric so that a higher number still means a better model. ^[1] The same pattern produces 'neg_log_loss', 'neg_brier_score', 'neg_mean_absolute_error', and friends.

Common scoring metrics

The table below lists frequently used scoring strings in scikit-learn alongside their typical use cases. ^[1]

Scroll sideways for more →

For custom needs, sklearn.metrics.make_scorer wraps any metric function into a callable scorer that obeys the higher-is-better convention. By default make_scorer assumes greater_is_better=True; for functions ending in _error, _loss, or _deviance, which return a value to minimize, you set greater_is_better=False and the scorer negates the result. ^[3]

How do you choose a scoring metric?

The choice of scoring metric is rarely neutral. Accuracy is the default for classification but quietly fails on imbalanced data. A model that always predicts "no fraud" on a dataset where fraud is one percent of transactions scores 99 percent accuracy and detects zero fraud. ROC AUC sidesteps that by measuring how well the model ranks positives above negatives at every threshold, which is why it dominates as a default for binary classification with skewed classes. Average precision (the area under the precision-recall curve) tends to be more informative than ROC AUC when the positive class is very rare, because precision-recall focuses on the region of the threshold space where positives live.

For regression, R-squared is comparable across problems but can be misleading when the target variance differs between training and test sets. RMSE and MAE are in the same units as the target, which makes them easier to communicate. MAE is more robust to outliers; RMSE penalizes large errors more heavily. Probabilistic predictions need probabilistic metrics. If a downstream decision uses the probability directly (for example, an insurance premium that scales with predicted loss probability), then log loss or Brier score is the right evaluation target rather than accuracy. ^[1]

Explain like I'm 5 (ELI5)

Scoring is the part where the model actually gives you an answer. You feed in a new example, the model spits out a number, and that number tells you what it thinks. Sometimes the number is a probability, like "there is a 73 percent chance this email is spam." Sometimes it is a ranking, like "this video is a better match for you than that one." Sometimes it is a credit score, like 720 out of 850. The same word, scoring, is also used to grade the model itself. You give the model a test, compare its answers to the real ones, and the test result is a score that tells you how good the model is at its job.

References

1. scikit-learn developers. "3.4. Metrics and scoring: quantifying the quality of predictions." scikit-learn documentation. https://scikit-learn.org/stable/modules/model_evaluation.html ↩
2. scikit-learn developers. "1.16. Probability calibration." scikit-learn documentation. https://scikit-learn.org/stable/modules/calibration.html ↩
3. scikit-learn developers. "sklearn.metrics.make_scorer." scikit-learn documentation. https://scikit-learn.org/stable/modules/generated/sklearn.metrics.make_scorer.html ↩
4. Google for Developers. "Scoring | Recommendation Systems." Machine Learning Crash Course. https://developers.google.com/machine-learning/recommendation/dnn/scoring ↩
5. Google for Developers. "Candidate generation overview." Machine Learning Crash Course. https://developers.google.com/machine-learning/recommendation/overview/candidate-generation ↩
6. Covington, Paul, Jay Adams, and Emre Sargin. "Deep Neural Networks for YouTube Recommendations." Proceedings of the 10th ACM Conference on Recommender Systems, 2016, pp. 191-198. https://research.google/pubs/deep-neural-networks-for-youtube-recommendations/ ↩
7. Wikipedia. "Learning to rank." https://en.wikipedia.org/wiki/Learning_to_rank ↩
8. Wikipedia. "Brier score." https://en.wikipedia.org/wiki/Brier_score ↩
9. FICO. "Can Machine Learning Build a Better FICO Score?" FICO Blog. https://www.fico.com/blogs/can-machine-learning-build-better-fico-score ↩
10. FICO. "Machine Learning and FICO Scores." FICO white paper. https://www.fico.com/en/latest-thinking/white-paper/machine-learning-and-fico-scores ↩
11. Wikipedia. "VantageScore." https://en.wikipedia.org/wiki/VantageScore ↩
12. Microsoft Learn. "Deploy models for scoring in batch endpoints." Azure Machine Learning documentation. https://learn.microsoft.com/en-us/azure/machine-learning/how-to-use-batch-model-deployments ↩