Estimator

See also: Machine learning terms

An estimator is a rule, function, or algorithm that takes observed data and produces a value intended to approximate some unknown quantity, typically a parameter of a probability distribution or a function relating inputs to outputs. The single value an estimator returns for a given dataset is called an estimate, and the unknown quantity being targeted is the estimand. The term carries two related but distinct meanings in modern usage. In classical statistics, an estimator is a procedure for inferring population parameters from a sample. In machine learning libraries such as scikit-learn, the word also names a software object that fits a model to data and exposes methods like fit and predict. The two senses are connected because library estimators perform parameter estimation under the hood, often using techniques developed by statisticians a century ago.

In Larry Wasserman's textbook All of Statistics, the statistical sense is summarized in one line: "Point estimation refers to providing a single 'best guess' of some quantity of interest," where the quantity "could be a parameter in a parametric model, a CDF F, a probability density function f, a regression function r, or a prediction for a future value Y of some random variable" ^[22]. Formally, given data points $X_1, \ldots, X_n$ drawn from a distribution $F$, "a point estimator $\hat\theta_n$ of a parameter $\theta$ is some function of $X_1, \ldots, X_n$" ^[22]. Because the estimator is a function of random data it is itself a random variable, while the parameter $\theta$ it targets is "a fixed, unknown quantity" ^[22].

The formal study of estimators began with Ronald A. Fisher's 1922 paper On the Mathematical Foundations of Theoretical Statistics, which introduced consistency, efficiency, sufficiency, and the method of maximum likelihood as a unified framework for evaluating estimation procedures ^[1]. The same paper introduced the now-standard terms "statistic," "parameter," and "maximum likelihood estimate" ^[2]. Earlier work by Carl Friedrich Gauss, Adrien-Marie Legendre, and Karl Pearson had produced specific estimation techniques, but Fisher's paper gave the field its theoretical vocabulary ^[2]. Subsequent contributions from Harald Cramér ^[3], C. R. Rao ^[4], Charles Stein ^[7], and Peter Huber ^[5] extended the theory in directions that shaped both classical and modern statistical practice.

What is the history of estimation theory?

The earliest formal estimators predate the word itself. Adrien-Marie Legendre published the method of least squares in 1805, applying it to the shape of the Earth. Gauss claimed to have used the method since 1795 and famously deployed it in 1801 to predict the orbit of the newly discovered asteroid Ceres after Italian astronomer Giuseppe Piazzi lost track of it. Gauss connected least squares to the normal distribution and probability theory in his 1809 Theoria Motus, transforming what Legendre had presented as an algebraic curve-fitting trick into a statistical method ^[9]. The priority dispute between Gauss and Legendre over who invented least squares became one of the most famous controversies in the history of statistics ^[9].

Karl Pearson introduced the method of moments in 1894, applying it to fit skewed distributions to data on crab measurements ^[8]. Pearson borrowed the term "moments" from physics and used the technique to estimate the parameters of his Pearson family of distributions, since maximum likelihood was not yet available ^[12].

Fisher's 1922 paper is generally regarded as the founding document of modern estimation theory ^[1][2]. He introduced maximum likelihood estimation as a general principle, defined what it means for an estimator to be consistent and efficient, and proved that the sufficient statistic for a normal scale parameter is the sum of squared deviations rather than the sum of absolute deviations championed by astronomer Arthur Eddington ^[1]. The paper also introduced the concept of Fisher information, which measures how much a sample tells us about an unknown parameter. Fisher had first used the phrase "method of maximum likelihood" in a 1912 paper, but it was the 1922 work that placed it on a rigorous footing ^[2].

In 1945, C. R. Rao published a lower bound on the variance of any unbiased estimator ^[4], derived independently and almost simultaneously by Harald Cramér in his 1946 textbook Mathematical Methods of Statistics ^[3]. The result, now called the Cramer-Rao bound, states that no unbiased estimator can have variance below the inverse of the Fisher information ^[10]. The story goes that Rao, then 24 years old and teaching estimation at Calcutta University, was asked by a student why a Fisher result for large samples could not be proved for finite samples. Rao went home, worked through the night, and produced the inequality the next day.

Charles Stein shocked the statistics community in 1956 by showing that the sample mean is inadmissible as an estimator of the mean vector of a multivariate normal distribution when the dimension is three or more ^[7]. Willard James and Stein produced an explicit dominating estimator in 1961, now called the James-Stein estimator ^[6]. The result is paradoxical because it implies that one can do better at estimating, say, the batting averages of three baseball players by shrinking each estimate toward a common value, even though the players have nothing to do with each other ^[11].

Peter Huber's 1964 paper Robust Estimation of a Location Parameter, published in the Annals of Mathematical Statistics, founded the field of robust statistics by introducing M-estimators ^[5]. These estimators interpolate between the sample mean and the sample median, and they remain useful when the data contains outliers or when the assumed distribution is only approximately correct ^[5].

What is a statistical estimator?

Formally, an estimator is a measurable function from the sample space to the parameter space. If the data is a random sample $X_1, X_2, \ldots, X_n$ drawn from a distribution with unknown parameter $\theta$, an estimator $\hat\theta$ is any function $\hat\theta = T(X_1, \ldots, X_n)$ that returns a value in the same space as $\theta$. The estimator is itself a random variable because it depends on random data; the value it returns for a particular sample is called an estimate.

A crucial distinction separates point estimators from interval estimators. A point estimator returns a single value, such as the sample mean as an estimate of the population mean. An interval estimator returns a range, such as a 95% confidence interval, that is intended to contain the true parameter with a specified probability. Confidence intervals were developed by Jerzy Neyman in 1937 as a frequentist counterpart to Bayesian credible intervals ^[17]. The frequentist interpretation is subtle and often misunderstood: a 95% confidence interval does not mean the true parameter lies in the computed interval with probability 0.95. As Wasserman puts it bluntly, "$C_n$ is random and $\theta$ is fixed" ^[22]. It means that if the estimation procedure were repeated on many independent samples, about 95% of the resulting intervals would contain the true value. The parameter is fixed, and the interval is the random quantity ^[17].

What makes a good estimator?

Statisticians evaluate estimators against a small set of formal properties. No single estimator is best on all criteria; choice depends on the application and on what one is willing to assume about the data.

Scroll sideways for more →

Fisher's 1922 definitions remain the working vocabulary. He described an efficient estimator as one for which "in large samples, when the distributions of the statistics tend to normality, that statistic is to be chosen which has the least probable error" ^[1][2], and an estimator is sufficient when it "contains the whole of the information" about the parameter that the sample provides ^[2]. Modern practice still tests estimators against exactly these criteria.

The mean squared error decomposition combines bias and variance into a single criterion. Wasserman states it as a theorem: the MSE "can be written as $\text{MSE} = \text{bias}^2(\hat\theta_n) + \mathbb{V}(\hat\theta_n)$" ^[22]:

$\text{MSE}(\hat\theta) = E[(\hat\theta - \theta)^2] = \text{Bias}(\hat\theta)^2 + \text{Var}(\hat\theta)$

This decomposition motivates the bias-variance tradeoff that pervades modern machine learning. A slightly biased estimator can have lower MSE than the best unbiased one, and the James-Stein estimator is the canonical example ^[11]. Ridge regression and most regularization techniques exploit this tradeoff explicitly. A direct corollary is that if both the bias and the standard error of an estimator tend to zero as the sample size grows, the estimator is consistent ^[22].

What is minimum-variance unbiased estimation?

If one restricts attention to unbiased estimators, a natural goal is to find the one with the smallest variance. An estimator that achieves the lowest variance among all unbiased estimators, for every value of the parameter, is called a minimum-variance unbiased estimator (MVUE), sometimes written UMVUE for uniformly minimum variance ^[18]. The MVUE is closely related to efficiency but is not the same thing. An efficient estimator, one whose variance meets the Cramer-Rao bound, is automatically the MVUE when it exists, yet many problems admit an MVUE that does not attain the bound. Efficiency is the stronger condition ^[18].

Two classical theorems explain how to construct the MVUE. The Rao-Blackwell theorem says that if you take any unbiased estimator and condition it on a sufficient statistic, the resulting estimator is still unbiased and has variance no larger than the original. In symbols, if $\delta$ is unbiased for $g(\theta)$ and $T$ is sufficient, then $\eta = E[\delta \mid T]$ is unbiased and $\text{Var}(\eta) \leq \text{Var}(\delta)$ ^[18]. Conditioning on a sufficient statistic can only help, never hurt. The Lehmann-Scheffé theorem sharpens this: if $T$ is both sufficient and complete, then any unbiased function of $T$ is the unique MVUE ^[18]. Together the two results give a recipe. Find a complete sufficient statistic, find any unbiased estimator that depends on the data only through it, and you are done.

What are the asymptotic properties of estimators?

Many estimators cannot be evaluated exactly for finite samples, so statisticians study how they behave as the sample size $n$ grows without bound. The central large-sample result concerns maximum likelihood. Under standard regularity conditions, the maximum likelihood estimator is consistent and asymptotically normal: the rescaled error $\sqrt{n},(\hat\theta - \theta)$ converges in distribution to a normal distribution with mean zero and variance equal to the inverse Fisher information $I(\theta)^{-1}$ ^[19]. Two facts fall out of this. First, the MLE converges at the rate $1/\sqrt{n}$, which is the usual rate for well-behaved parametric estimators. Second, its asymptotic variance equals the Cramer-Rao lower bound, so the MLE is asymptotically efficient: no consistent estimator does better in the limit ^[19][10]. These guarantees hold only when the regularity conditions are met. They can fail when the parameter sits on the boundary of its space, when the support of the distribution depends on the parameter, or when the model is misspecified ^[19].

A handful of estimation principles dominate practice. Each can be derived from a different criterion, and each has characteristic strengths.

Scroll sideways for more →

Maximum likelihood is the default in much of modern statistics because it is asymptotically efficient under mild conditions: as the sample size grows, no other consistent estimator achieves lower variance ^[1][19]. The method also generalizes naturally to complex models, including the deep neural networks trained today by minimizing negative log-likelihood (cross-entropy) loss. Ordinary least squares is itself a maximum likelihood estimator when the errors are assumed to be independent and normally distributed, which is one reason the two methods so often agree in practice ^[13].

What is the difference between parametric and non-parametric estimators?

The distinction between parametric and non-parametric estimation runs through the entire field.

A parametric model assumes the data was generated by a distribution that belongs to a family indexed by a finite-dimensional parameter. The job of the estimator is to pin down that parameter. Linear regression, logistic regression, Gaussian mixture models, and most classical statistical methods are parametric. The advantage is statistical efficiency: when the model is correct, parametric estimators converge fast and produce tight confidence intervals. The risk is misspecification, since a wrong family produces misleading results no matter how much data is available. Buja and colleagues argued in 2019 that classical inference quietly assumes the model is exactly right, and that once a regression model is treated as an approximation rather than the truth, the standard errors and confidence intervals it produces can be badly off ^[16].

A non-parametric model does not commit to a fixed parameter count. The number of effective parameters often grows with the sample size. Kernel density estimation, k-nearest neighbors, decision trees, and Gaussian processes are non-parametric. These methods make weaker assumptions about the data-generating process and are more flexible, but they typically converge slower and require more data to reach a given accuracy.

Popular parametric estimators in machine learning include:

• Linear regression, which assumes the relationship between dependent and independent variables can be described by a linear equation with two parameters (slope and intercept) in the simple case.
• Logistic regression, which models the probability of a binary outcome by assuming a linear relationship between the predictors and the log-odds.
• Gaussian mixture models, fit by expectation-maximization, which assume the data is drawn from a finite weighted sum of normal distributions.

Popular non-parametric estimators include:

• K-nearest neighbors (KNN), which estimates the target value at a query point by averaging or voting over the labels of the $k$ closest training points in the feature space.
• Kernel density estimation (KDE), which estimates a probability density by placing a kernel function (often Gaussian) at each observation and summing the kernels into a smooth curve.
• Decision trees and random forests, which partition the input space recursively without committing to a global functional form.

In practice, the line is blurry. Many models are semi-parametric, combining a finite-dimensional parameter of interest with an infinite-dimensional nuisance component. The Cox proportional hazards model is a classic example.

What is an estimator in scikit-learn?

The word estimator has a specific technical meaning in scikit-learn, the dominant Python machine learning library, and by extension across many libraries that adopt its API conventions. The scikit-learn documentation states the definition plainly: "The API has one predominant object: the estimator. An estimator is an object that fits a model based on some training data and is capable of inferring some properties on new data" ^[14]. The conventions were first codified in a 2013 design paper by the scikit-learn developers, which argued that a single uniform estimator interface is what lets the library compose and reuse otherwise unrelated algorithms ^[20]. That paper describes the estimator interface as "the most important" of the library's three core interfaces and notes that it "defines instantiation mechanisms of objects and exposes a fit method for learning a model from training data" ^[20].

A scikit-learn estimator is any object that implements a fit method to learn from data ^[14]. The base class sklearn.base.BaseEstimator provides the common machinery for getting and setting hyperparameters, for the textual and HTML representation shown in notebooks, and for cloning and serialization ^[15]. By convention, an estimator object stores nothing in its __init__: hyperparameters are accepted but not validated, and no learned state is created until fit is called ^[14]. Every hyperparameter must be saved to an attribute of the same name, with no renaming and no *args or **kwargs, because tools like clone and GridSearchCV rely on reading those attributes back out ^[14]. After fitting, the estimator stores learned parameters in attributes whose names end with an underscore (such as coef_ for linear models or cluster_centers_ for KMeans). The documentation makes this convention explicit: "attributes which you'd want to expose to your users as public attributes and have been estimated or learned from the data must always have a name ending with trailing underscore" ^[14]. This trailing-underscore convention is the visible signal that an attribute exists only after fit ^[14].

Two methods inherited from BaseEstimator make hyperparameters introspectable. get_params returns the constructor arguments as a dictionary, and set_params writes them back, accepting the nested <component>__<parameter> syntax that lets a search reach inside a pipeline ^[15]. The related function sklearn.base.clone builds a fresh, unfitted copy of an estimator from those same parameters, which is how cross-validation gets an untrained model for each fold ^[14]. By the same logic, fit is required to return self, so calls can be chained and so clone can reproduce the object; the documentation notes "the method should return the object (self)" precisely so one-liners like SGDClassifier(alpha=10).fit(X_train, y_train).predict(X_test) work ^[14]. When an estimator is fitted on tabular data it records the input width in n_features_in_, and if it was fitted on a pandas or polars DataFrame it also records the column names in feature_names_in_, then checks both at prediction time to catch shape and naming mistakes ^[14].

What are the categories of scikit-learn estimators?

Scikit-learn estimators specialize into a small number of categories, each defined by an additional mixin class ^[14].

Scroll sideways for more →

Each mixin supplies more than a label. RegressorMixin and ClassifierMixin both provide a default score method, but they score differently: a regressor reports the coefficient of determination $R^2$, while a classifier reports mean accuracy ^[14]. Classifiers may additionally expose predict_proba for calibrated class probabilities and decision_function for the raw signed distance from the decision boundary, both keyed to the class labels recorded in the classes_ attribute during fit ^[14]. TransformerMixin supplies fit_transform, which fits and transforms in one pass and is often faster than calling the two steps separately ^[14].

Because the categories are conventions rather than strict types, scikit-learn leans on duck typing instead of inheritance checks. Helper functions such as sklearn.base.is_classifier and is_regressor inspect an object for the right behavior rather than testing its class, so an estimator that follows the protocol works even if it does not inherit from the canonical mixins ^[14]. The uniformity of the API has practical consequences. Because every estimator follows the same conventions, scikit-learn can compose them into pipelines, run cross-validation over arbitrary models, and search hyperparameter spaces with GridSearchCV or RandomizedSearchCV. The same loop that trains a logistic regression also trains a gradient boosting model, since both expose fit and predict.

How do pipelines and meta-estimators work?

A Pipeline chains transformers and a final estimator into a single object that itself looks like an estimator. Calling fit on the pipeline calls fit_transform on each transformer in sequence and fit on the final estimator. This pattern prevents data leakage, since transformations are learned on training folds and applied to test folds without contamination.

Meta-estimators wrap other estimators to add behavior. GridSearchCV wraps an estimator to perform exhaustive hyperparameter search via cross-validation. BaggingClassifier wraps a base classifier to produce an ensemble. MultiOutputRegressor wraps a single-output regressor to handle multi-output targets. Because all of these objects respect the estimator interface, they nest arbitrarily.

What is incremental and out-of-core learning?

Not every dataset fits in memory, and not every problem allows a single pass over the data. Estimators that support incremental learning expose a partial_fit method that updates the model from one mini-batch at a time and can be called repeatedly ^[21]. This is the basis for out-of-core learning, in which the data is streamed from disk in chunks so that only a small batch is ever resident in RAM ^[21]. SGDClassifier and SGDRegressor, the naive Bayes variants, and MiniBatchKMeans are common examples that implement partial_fit ^[21]. A related but distinct mechanism is the warm_start parameter. When it is set to true, calling fit again reuses the existing solution as a starting point instead of discarding it, which is useful for fitting a model along a path of growing complexity such as adding trees to a gradient boosting ensemble ^[21].

How are scikit-learn estimators tested for conformance?

Because so much downstream code assumes the estimator contract, scikit-learn ships a conformance test suite. The function check_estimator runs a battery of checks against an estimator instance, and the parametrize_with_checks decorator turns those checks into individual pytest cases ^[14]. Estimators advertise their capabilities through tags, exposed via the __sklearn_tags__ method, which tell the checker things like whether the estimator handles sparse input, allows missing values, or is non-deterministic, so that only the relevant tests run ^[14]. This machinery is what lets third-party libraries build estimators that drop into scikit-learn pipelines and cross-validation without modification.

Example

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV

pipe = Pipeline([
    ("scaler", StandardScaler()),
    ("clf", LogisticRegression(max_iter=1000)),
])

params = {"clf__C": [0.01, 0.1, 1.0, 10.0]}
search = GridSearchCV(pipe, params, cv=5)
search.fit(X_train, y_train)
print(search.best_params_, search.score(X_test, y_test))

The pipeline is itself an estimator, the grid search is an estimator wrapping that estimator, and the whole stack exposes fit, predict, and score. The composability comes directly from the discipline of the base interface.

How are the two senses of estimator connected?

The statistical and software senses of estimator are linked by what happens inside fit. When LinearRegression.fit runs, it computes the ordinary least squares solution that Gauss and Legendre worked out two centuries ago. When LogisticRegression.fit runs, it computes a maximum likelihood estimate of the coefficients, typically via the Newton-Raphson or L-BFGS algorithm. When GaussianMixture.fit runs, it iterates the Expectation-Maximization algorithm to find an MLE of the mixture parameters. When BayesianRidge.fit runs, it computes a Bayes estimator under a specific prior.

The properties statisticians study, including consistency, unbiasedness, and efficiency, apply directly to these procedures. Cross-validation, hyperparameter tuning, and regularization are all techniques that trade bias for variance in pursuit of low test error, which is the bias-variance tradeoff by another name. The library API hides the mathematics behind a uniform interface, but the underlying concepts are the same ones Fisher named in 1922 ^[1].

Explain like I'm 5

Imagine you have a jar of jellybeans and you want to guess how many are inside without dumping them out. You could grab a small handful, count them, and multiply. That counting-and-multiplying procedure is an estimator. Different procedures might be more accurate (count two handfuls and average), more cautious (only count handfuls of a certain size), or more honest about their uncertainty (give a range instead of one number). In statistics, we study which procedures give the best guesses on average, which ones get closer to the truth as you grab more jellybeans, and which ones are not fooled when somebody slipped marbles into the jar.

References

1. Fisher, R. A. (1922). On the mathematical foundations of theoretical statistics. *Philosophical Transactions of the Royal Society A*, 222, 309-368. https://doi.org/10.1098/rsta.1922.0009 ↩
2. Aldrich, J. (2015). From evidence to understanding: a commentary on Fisher (1922). *Philosophical Transactions of the Royal Society A*, 373(2039). https://royalsocietypublishing.org/doi/10.1098/rsta.2014.0252 ↩
3. Cramér, H. (1946). *Mathematical Methods of Statistics*. Princeton University Press. ↩
4. Rao, C. R. (1945). Information and the accuracy attainable in the estimation of statistical parameters. *Bulletin of the Calcutta Mathematical Society*, 37, 81-91. ↩
5. Huber, P. J. (1964). Robust estimation of a location parameter. *The Annals of Mathematical Statistics*, 35(1), 73-101. https://doi.org/10.1214/aoms/1177703732 ↩
6. James, W., and Stein, C. (1961). Estimation with quadratic loss. *Proceedings of the Fourth Berkeley Symposium on Mathematical Statistics and Probability*, 1, 361-379. ↩
7. Stein, C. (1956). Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. *Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability*, 1, 197-206. ↩
8. Pearson, K. (1894). Contributions to the mathematical theory of evolution. *Philosophical Transactions of the Royal Society A*, 185, 71-110. ↩
9. Stigler, S. M. (1981). Gauss and the invention of least squares. *The Annals of Statistics*, 9(3), 465-474. ↩
10. Cramér-Rao bound. *Wikipedia*. https://en.wikipedia.org/wiki/Cram%C3%A9r%E2%80%93Rao_bound ↩
11. James-Stein estimator. *Wikipedia*. https://en.wikipedia.org/wiki/James%E2%80%93Stein_estimator ↩
12. Method of moments (statistics). *Wikipedia*. https://en.wikipedia.org/wiki/Method_of_moments_(statistics) ↩
13. Least squares. *Wikipedia*. https://en.wikipedia.org/wiki/Least_squares ↩
14. scikit-learn developers. (2024). Developing scikit-learn estimators. https://scikit-learn.org/stable/developers/develop.html ↩
15. scikit-learn developers. (2024). `sklearn.base.BaseEstimator` API reference. https://scikit-learn.org/stable/modules/generated/sklearn.base.BaseEstimator.html ↩
16. Buja, A., et al. (2019). Models as approximations: a conspiracy of random regressors and model deviations against classical inference in regression. *Statistical Science*, 34(4), 545-565. ↩
17. Neyman, J. (1937). Outline of a theory of statistical estimation based on the classical theory of probability. *Philosophical Transactions of the Royal Society A*, 236(767), 333-380. https://doi.org/10.1098/rsta.1937.0005 ↩
18. Minimum-variance unbiased estimator. *Wikipedia*. https://en.wikipedia.org/wiki/Minimum-variance_unbiased_estimator ↩
19. Maximum likelihood estimation. *Wikipedia*. https://en.wikipedia.org/wiki/Maximum_likelihood_estimation ↩
20. Buitinck, L., et al. (2013). API design for machine learning software: experiences from the scikit-learn project. *ECML PKDD Workshop: Languages for Data Mining and Machine Learning*, 108-122. https://arxiv.org/abs/1309.0238 ↩
21. scikit-learn developers. (2024). Strategies to scale computationally: bigger data. https://scikit-learn.org/stable/computing/scaling_strategies.html ↩
22. Wasserman, L. (2004). *All of Statistics: A Concise Course in Statistical Inference*. Springer. Chapter 6, "Models, Statistical Inference and Learning," pp. 87-92. ↩