Citation and evidence

Confidence interval

17 min full readUpdated 24 references

This article's verification

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

Machine LearningModel EvaluationStatistics

Cite this article

A confidence interval (CI) is a range of values, computed from sample data, that is produced by a procedure designed to contain the true value of an unknown parameter in a stated proportion of repeated samples. If 95 percent confidence intervals are calculated over and over in valid applications, about 95 percent of them will cover the true value; this long-run frequency is the interval's coverage probability, and it describes the procedure rather than any single interval.[1][2] Confidence intervals are a central tool of frequentist statistics, and they are closely tied to hypothesis testing: the values not rejected by a test at the 5 percent level typically form a 95 percent interval.[1] In machine learning, they are used to put error bars on benchmark scores, leaderboard ratings and other evaluation results that are estimated from a finite set of test items or training runs.[13][20][21]

Fifty vertical blue bars of varying position around a dashed horizontal line labelled mu
Fifty realisations of a 95% confidence interval for a mean mu (dashed line): most of the intervals cross the true value and a few miss it. Tsyplakov, via Wikimedia Commons, Public domain.

Definition

Let $\theta$ be an unknown parameter, such as a population mean or a model's true accuracy, and let $X$ be the observed data. A $100(1-\alpha)$ percent confidence interval is a pair of statistics $L(X)$ and $U(X)$ chosen so that

Pθ(L(X)≤θ≤U(X))=1−αP_\theta\big(L(X) \le \theta \le U(X)\big) = 1 - \alpha

for every possible value of $\theta$, at least approximately. The quantity $1-\alpha$ is called the confidence coefficient or confidence level, and $\alpha$ is the corresponding significance level.[2] Common choices are 90, 95 and 99 percent.[2] Exact methods such as the Clopper-Pearson interval for a proportion guarantee coverage of at least $1-\alpha$, though they tend to be conservative; most approximate methods aim for coverage close to $1-\alpha$ on average but can fall short in some cases.[5]

The probability statement refers to the random interval before the data are seen. Once a particular interval has been computed, it either contains the true value or it does not, so a computed 95 percent interval does not have a 95 percent probability of containing the parameter in the frequentist sense.[1][2]

Width

The width of an interval reflects how precisely the data pin down the parameter. For the standard interval for a mean, the width shrinks as the sample size grows (the sample size enters under a square root) and grows with the spread of the data.[2] Because the dependence is on $\sqrt{N}$, halving the width requires roughly four times as many observations. A higher confidence level uses a larger critical value from the reference distribution and so gives a wider interval for the same data.

"Confidence interval simulation | Confidence intervals | AP Statistics | Khan Academy", video from Khan Academy, published December 14, 2017. Watch on YouTube.

Constructing confidence intervals

Intervals for a mean

The textbook interval for the mean of a roughly normally distributed quantity is

Yˉ±t1−α/2, N−1 sN\bar{Y} \pm t_{1-\alpha/2,\,N-1}\,\frac{s}{\sqrt{N}}

where $\bar{Y}$ is the sample mean, $s$ the sample standard deviation, $N$ the sample size and $t_{1-\alpha/2,,N-1}$ the corresponding percentile of Student's t distribution with $N-1$ degrees of freedom.[2] The term $s/\sqrt{N}$ is the standard error of the mean. For large samples, the t percentile is replaced by a normal quantile, which gives the familiar 95 percent interval of the estimate plus or minus 1.96 standard errors.[13]

Intervals for a proportion

Many quantities in statistics and machine learning are proportions: the fraction of questions a model answers correctly, a defect rate, a click-through rate. The simplest interval, often called the Wald or normal-approximation interval, is

p^±z1−α/2p^(1−p^)n\hat{p} \pm z_{1-\alpha/2}\sqrt{\frac{\hat{p}(1-\hat{p})}{n}}

where $\hat{p}$ is the observed proportion and $n$ the number of trials.[3] It is the most frequently used formula, but its lower limit can be negative, which is impossible for a proportion.[3] Lawrence D. Brown, T. Tony Cai and Anirban DasGupta showed in 2001 that the coverage of the Wald interval is erratic, that its problems persist far more widely than commonly appreciated, and that common textbook advice about when it is safe to use is misleading.[4]

Several alternatives are in common use:

MethodConstructionNotes
Wald (normal approximation)Estimate plus or minus z standard errorsSimple; erratic coverage; can leave [0, 1][3][4]
Wilson scoreInverts the large-sample test for a proportionStays inside [0, 1]; recommended by Brown, Cai and DasGupta for small n[3][4]
Agresti-CoullAdjusted Wald intervalRecommended by Brown, Cai and DasGupta for larger n[4][5]
JeffreysEqual-tailed interval based on the Jeffreys priorRecommended by Brown, Cai and DasGupta for small n[4][5]
Clopper-PearsonExact interval from binomial tail probabilitiesCoverage at least $1-\alpha$, but generally conservative (wide)[3][5]

Expanded article table

The Wilson interval dates to a 1927 paper by Edwin B. Wilson, and the exact interval is named after C. J. Clopper and E. S. Pearson, whose 1934 paper in Biometrika illustrated confidence limits with the binomial case.[3][6][24] The US National Institute of Standards and Technology (NIST) notes that when the number of failures or the sample size is very small, normal-approximation limits may not be accurate enough for some applications, and gives the exact binomial interval for that case.[3] The statsmodels library implements all of these through its proportion_confint function, whose default method is the normal approximation; its documentation notes that the Clopper-Pearson interval is generally conservative and that most other methods can undercover in some cases.[5]

Bootstrap intervals

Many statistics have no convenient formula for their sampling distribution: medians, ratios, corpus-level metrics such as BLEU, or ratings fitted by a model. Bradley Efron introduced the bootstrap in 1979 as a general method for estimating the sampling distribution of a statistic from the observed data alone, by resampling the data with replacement.[7] Confidence intervals can then be read from the distribution of the statistic over many resamples.

The SciPy function scipy.stats.bootstrap illustrates the common variants. The percentile method takes the central portion of the bootstrap distribution that contains the requested confidence level; the basic (or "reverse percentile") method and the bias-corrected and accelerated (BCa) method differ only in how that interval is determined. SciPy's defaults are the BCa method, 9,999 resamples and a 95 percent confidence level, and its documentation describes the plain percentile method as intuitive but rarely used in practice.[8]

Relationship to hypothesis tests

Confidence intervals and significance tests are two views of the same calculation. If a test is run for every candidate value of the parameter, the values whose test gives a p-value above 0.05 typically form a 95 percent confidence interval, provided the statistical model is correct.[1] An interval that excludes zero for a difference between two groups therefore corresponds to a difference that is significant at the matching level. NIST derives the Wilson interval in exactly this way, by inverting the hypothesis test for a proportion.[3]

History

The calculation behind a modern confidence interval is older than the name. The statistician Glenn Shafer notes that Joseph Fourier's 1826 memoir called it "a 19 out of 20 bet" that an error would not exceed a stated bound, which corresponds to the familiar 95 percent interval of 1.96 standard errors, and that Antoine Augustin Cournot defined what is now called a confidence interval in his 1843 book on probability while explicitly rejecting Laplace's Bayesian interpretation.[9]

The modern theory and terminology come from Jerzy Neyman. According to Shafer, Neyman introduced the English terms "confidence coefficient" and "confidence interval" in a 1934 paper on survey sampling in the Journal of the Royal Statistical Society, and later explained (in a 1941 paper) that he had used such intervals in Poland since 1930 under the Polish name przedzial ufnosci.[9] Ronald Fisher had published intervals for parameters obtained without prior probabilities from 1930 onward, which he called fiducial intervals; Neyman at first presented his own definition as a clarification of Fisher's work, but Fisher was unhappy with this, and Neyman was pushed to present his theory as different from Fisher's.[9] Neyman's full treatment appeared in 1937 as "Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability" in the Philosophical Transactions of the Royal Society.[9][10] In it he observed that statistical publications rarely reported an estimate without an accompanying plus-or-minus term (its standard error $S_T$), and argued that statisticians in practice had in mind not a single estimate but a lower and an upper limit, set by constant multiples of $S_T$, between which the true value of the parameter presumably falls.[9]

Interpretation and misinterpretation

Confidence intervals are widely misread. A 2016 guide by Sander Greenland, Stephen Senn, Kenneth Rothman, John Carlin, Charles Poole, Steven Goodman and Douglas Altman lists common misinterpretations of statistical tests, confidence intervals and power, including these about intervals:[1]

Common claimWhy it is wrong
A specific 95% interval has a 95% chance of containing the true valueCoverage is a property of the procedure over many repetitions; a given interval either contains the value or does not
A value outside the 95% interval has been refuted by the dataThe interval depends on model assumptions that may be violated
If two intervals overlap, the difference between the estimates is not significantIntervals can overlap substantially while the test of the difference gives P < 0.05
95% of future estimates will fall inside the observed intervalThe chance is usually well below 95%; in the guide's example of two independent studies it is 83%
The interval that excludes the null is the more precise onePrecision is measured by width, not by whether the null value is excluded

Expanded article table

In the overlap example, intervals of (1.04, 4.96) and (4.16, 19.84) overlap, yet the test for a difference between the two estimates gives P = 0.03. Non-overlapping intervals do imply a significant difference, but the reverse does not hold.[1] The practical consequence for comparisons, including comparisons of machine learning models, is that the uncertainty of a difference should be computed directly rather than judged by eye from two separate error bars.

The misreadings are common even among professionals. In a study published in 2014, Rink Hoekstra, Richard Morey, Jeffrey Rouder and Eric-Jan Wagenmakers asked 120 psychology researchers and 442 psychology students to judge six statements about a confidence interval. All six statements were false, yet both groups endorsed more than three on average, and researchers' self-reported statistical experience did not predict their performance.[11]

Some statisticians go further and question whether confidence interval theory supports the uses researchers put intervals to. Morey and colleagues argued in 2016 that intervals do not necessarily index the precision of an estimate or the plausibility of values within them, and that the confidence level need not indicate how plausible it is that the true value lies inside. They cautioned against relying on confidence interval theory to justify interval estimates and suggested using other theories of interval estimation instead.[12] One alternative is the Bayesian credible interval, which has a 95 percent posterior probability of containing the parameter; confidence intervals are commonly misread as if they were credible intervals.[18] (See Bayesian inference.)

Use in machine learning evaluation

Why benchmark scores need intervals

A benchmark score is an estimate. The test questions are a finite sample from a larger space of possible questions, and the model's own outputs may vary from run to run, so the same model would score differently on a different draw of questions or a different random seed. The Llama 3 technical report states the point directly: benchmark scores are estimates that "have variance because benchmark sets are finite samples drawn from some underlying distribution."[15]

Research on evaluation practice has repeatedly found this uncertainty ignored. A 2024 study by Lovish Madaan and colleagues measured several kinds of benchmark variance, including variance across training seeds, and noted that such variance, which decides whether differences between models are meaningful, is rarely quantified.[16] Xavier Bouthillier and co-authors found in 2021 that data sampling, parameter initialization and hyperparameter choice all have a marked effect on benchmark results, and analysed how comparison methods in common use cope with that variance.[17]

Standard errors for evaluation scores

In November 2024 Evan Miller of Anthropic published "Adding Error Bars to Evals", which treats language model evaluations as experiments whose questions are drawn from an unseen super-population.[13][14] For a score averaged over $n$ questions, the standard error of the mean is

SE=Var(s)n\mathrm{SE} = \sqrt{\frac{\mathrm{Var}(s)}{n}}

which reduces to $\sqrt{\bar{s}(1-\bar{s})/n}$ for right-or-wrong scoring, and a 95 percent interval is the mean plus or minus 1.96 standard errors.[13][14] For example, a model that scores 80 percent on 1,000 independent questions has a standard error of about 1.3 percentage points, giving a 95 percent interval of roughly 77.5 to 82.5 percent. The paper's recommendations are:[13]

RecommendationReason given
Compute standard errors using the central limit theoremApplies to any eval with finite-variance scores and many questions; the bootstrap is unnecessary unless the sampling scheme or estimator is complicated
Cluster standard errors when questions come in related groupsQuestions sharing a passage (as in DROP, SQuAD and RACE) or translated from one source (as in MGSM) are not independent; on DROP the clustered standard error was about three times the naive one
Reduce variance within questionsResample answers several times for chain-of-thought evals; use next-token probabilities when there is no chain of thought
Compare models on question-level paired differencesModels tend to agree on which questions are easy or hard, so pairing reduces the variance of the comparison at no extra cost
Use power analysisIn the paper's worked example, detecting a 3-point difference with 80 percent power at the 5 percent level needs about 969 questions; it suggests new evals contain at least 1,000 questions

Expanded article table

The paper also advises against lowering the sampling temperature to reduce variance, and notes that error bars "remain noticeably absent" from traditional question-and-answer evals, citing the Llama 3 report as a notable exception.[13] That report computed its 95 percent intervals as $1.96\sqrt{S(1-S)/N}$, where $S$ is the observed score and $N$ the benchmark size, while acknowledging that the Gaussian assumption is incorrect for bounded scores and that preliminary bootstrap experiments suggested it is a good approximation for discrete metrics.[15]

Small benchmarks

The normal approximation behind these formulas works poorly on small benchmarks. In a position paper presented as a spotlight at ICML 2025, Sam Bowyer, Laurence Aitchison and Desi Ivanova argued that intervals based on the central limit theorem are appropriate for benchmarks with thousands of examples but usually understate uncertainty, producing error bars that are too small, on the smaller and more specialized benchmarks that are increasingly common.[18] When a model gets every question right or every question wrong, the standard error is zero and the interval collapses to a point. In their simulations the Wilson score interval and simple Bayesian intervals based on a Beta-Bernoulli model achieved close to nominal coverage at small sample sizes, while the exact Clopper-Pearson interval was overly conservative; they recommended the Wilson or Bayesian intervals in practice.[18]

Bootstrap intervals in NLP

Bootstrap intervals are common for metrics that are not simple averages. Philipp Koehn's 2004 paper on significance tests for machine translation evaluation used bootstrap resampling to compute confidence intervals for BLEU, since BLEU is not the mean of sentence-level scores and has no simple analytic interval. Test sets of 300 sentences were resampled with replacement 1,000 times, and the 25 highest and 25 lowest scores were dropped to leave a 95 percent interval. Checked against a score computed on 30,000 sentences, the intervals from 100 test sets covered that near-true score in all but three cases.[19]

Leaderboards

Pairwise-preference leaderboards also report intervals. The 2024 paper describing Chatbot Arena fits Bradley-Terry coefficients to crowdsourced votes and considers two ways to put intervals on them: the pivot bootstrap and "sandwich" robust standard errors. The authors chose to deploy the sandwich intervals because they are smaller in large samples.[20] Because a ranking compares every model at once, the paper also builds simultaneous intervals with a multiplicity correction, and it assigns each model a rank of one plus the number of models whose lower bound lies above its upper bound, so that a model is ranked below another only when their intervals separate.[20]

Reinforcement learning

In deep reinforcement learning, training runs are expensive, and results are often reported from only a few runs per task. Rishabh Agarwal and colleagues showed in 2021, using the Atari 100k benchmark as a case study, that conclusions drawn from point estimates of aggregate performance can differ substantially from those of a fuller statistical analysis, and argued for reporting interval estimates. The paper received an Outstanding Paper Award at NeurIPS 2021.[21] Its accompanying open-source library, rliable, computes stratified bootstrap confidence intervals and promotes the interquartile mean, the average over the middle 50 percent of runs, as an aggregate that is less sensitive to outliers than the mean and more statistically efficient than the median.[22]

Prediction-powered inference

Machine learning predictions can also be used to make intervals narrower. Prediction-powered inference, introduced in 2023 by Anastasios Angelopoulos, Stephen Bates, Clara Fannjiang, Michael I. Jordan and Tijana Zrnic, combines a small labelled dataset with predictions from a machine learning system to produce provably valid confidence intervals for quantities such as means, quantiles and regression coefficients, without assumptions about the model that supplies the predictions. More accurate predictions give smaller intervals.[23]

Continue reading

More to read

Topics:Machine LearningModel EvaluationStatistics

References

  1. ^1 ^2 ^3 ^4 ^5 ^6Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., Altman, D. G. "Statistical tests, P values, confidence intervals, and power: a guide to misinterpretations". European Journal of Epidemiology 31(4):337-350, April 2016. pmc.ncbi.nlm.nih.gov/...PMC4877414
  2. ^1 ^2 ^3 ^4 ^5 ^6NIST/SEMATECH. "1.3.5.2. Confidence Limits for the Mean". e-Handbook of Statistical Methods. itl.nist.gov/...eda352
  3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8NIST/SEMATECH. "7.2.4.1. Confidence intervals". e-Handbook of Statistical Methods. itl.nist.gov/...prc241
  4. ^1 ^2 ^3 ^4 ^5Brown, L. D., Cai, T. T., DasGupta, A. "Interval Estimation for a Binomial Proportion". Statistical Science 16(2):101-133, 2001. doi.org/...1009213286
  5. ^1 ^2 ^3 ^4 ^5statsmodels developers. "statsmodels.stats.proportion.proportion_confint". statsmodels documentation. statsmodels.org/...s.proportion.proportion_confint
  6. ^Clopper, C. J., Pearson, E. S. "The Use of Confidence or Fiducial Limits Illustrated in the Case of the Binomial". Biometrika 26(4):404-413, 1934. doi.org/...26.4.404
  7. ^Efron, B. "Bootstrap Methods: Another Look at the Jackknife". The Annals of Statistics 7(1):1-26, 1979. doi.org/...1176344552
  8. ^SciPy developers. "scipy.stats.bootstrap". SciPy documentation. docs.scipy.org/...scipy.stats.bootstrap
  9. ^1 ^2 ^3 ^4 ^5Shafer, G. "On the nineteenth-century origins of significance testing and p-hacking". arXiv:2610.06233, October 2026. arxiv.org/...2610.06233
  10. ^Neyman, J. "Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability". Philosophical Transactions of the Royal Society of London. Series A, Mathematical and Physical Sciences 236(767):333-380, 1937. doi.org/...rsta.1937.0005
  11. ^Hoekstra, R., Morey, R. D., Rouder, J. N., Wagenmakers, E.-J. "Robust misinterpretation of confidence intervals". Psychonomic Bulletin & Review 21(5):1157-1164, 2014. research.rug.nl/...97d-c103-44c5-bb9d-3dccb9b102b7
  12. ^Morey, R. D., Hoekstra, R., Rouder, J. N., Lee, M. D., Wagenmakers, E.-J. "The fallacy of placing confidence in confidence intervals". Psychonomic Bulletin & Review 23(1):103-123, 2016. doi.org/...s13423-015-0947-8
  13. ^1 ^2 ^3 ^4 ^5 ^6Miller, E. "Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations". arXiv:2411.00640, November 2024. arxiv.org/...2411.00640
  14. ^1 ^2Anthropic. "A statistical approach to model evaluations". Anthropic, 2024-11-19. anthropic.com/...statistical-approach-to-model-evals
  15. ^1 ^2Llama Team, AI @ Meta. "The Llama 3 Herd of Models". arXiv:2407.21783, 2024. arxiv.org/...2407.21783
  16. ^Madaan, L., Singh, A. K., Schaeffer, R., Poulton, A., Koyejo, S., Stenetorp, P., Narang, S., Hupkes, D. "Quantifying Variance in Evaluation Benchmarks". arXiv:2406.10229, June 2024. arxiv.org/...2406.10229
  17. ^Bouthillier, X., Delaunay, P., Bronzi, M., et al. "Accounting for Variance in Machine Learning Benchmarks". arXiv:2103.03098, March 2021. arxiv.org/...2103.03098
  18. ^1 ^2 ^3Bowyer, S., Aitchison, L., Ivanova, D. R. "Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints". arXiv:2503.01747 (ICML 2025 spotlight position paper), 2025. arxiv.org/...2503.01747
  19. ^Koehn, P. "Statistical Significance Tests for Machine Translation Evaluation". Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pp. 388-395, July 2004. aclanthology.org/W04-3250
  20. ^1 ^2 ^3Chiang, W.-L., Zheng, L., Sheng, Y., Angelopoulos, A. N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., Gonzalez, J. E., Stoica, I. "Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference". arXiv:2403.04132, March 2024. arxiv.org/...2403.04132
  21. ^1 ^2Agarwal, R., Schwarzer, M., Castro, P. S., Courville, A., Bellemare, M. G. "Deep Reinforcement Learning at the Edge of the Statistical Precipice". arXiv:2108.13264 (NeurIPS 2021), 2021. arxiv.org/...2108.13264
  22. ^Google Research. "rliable". GitHub. github.com/...rliable
  23. ^Angelopoulos, A. N., Bates, S., Fannjiang, C., Jordan, M. I., Zrnic, T. "Prediction-Powered Inference". arXiv:2301.09633, January 2023. arxiv.org/...2301.09633
  24. ^Wilson, E. B. "Probable Inference, the Law of Succession, and Statistical Inference". Journal of the American Statistical Association 22(158):209-212, 1927. doi.org/...01621459.1927.10502953

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

2 revisions · v2 · 3,492 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check before publication on November 18, 2026: 78 claims checked against 24 cited and independent sources; 0 material and 5 minor corrections applied before publication.

Cite this page: AI Wiki. "Confidence interval." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/confidence_interval

Edit article