Citation and evidence

Conformal prediction

24 min full readUpdated 30 references

This article's verification

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

Machine LearningModel EvaluationStatistics

Cite this article

Conformal prediction is a family of statistical methods that turn the output of any predictive model into a prediction set or prediction interval that contains the true answer with a probability the user chooses, such as 90%, without assuming a particular data distribution or a correct model.[3] The only assumption behind the basic guarantee is that the calibration data and the new test point are exchangeable, a weaker condition than being independent and identically distributed.[2] The framework was developed by Vladimir Vovk, Alexander Gammerman and collaborators from the late 1990s and set out in the 2005 book Algorithmic Learning in a Random World by Vovk, Gammerman and Glenn Shafer.[1][3] It is also called conformal inference, and it has become a common tool for distribution-free uncertainty quantification in machine learning, including for deep learning classifiers[10] and, more recently, for large language models.[23]

The most widely used form is split conformal prediction, also called inductive conformal prediction, which needs only a trained model and a held-out calibration set.[3][5] It computes a score that measures how badly the model's prediction disagrees with the true label on each calibration example, takes an adjusted quantile of those scores, and includes in the prediction set every candidate answer whose score falls below that quantile.[3] The guarantee holds whatever the model is and however poorly it fits, but the usefulness of the sets (how small they are) depends on the model and the score function.[3]

The coverage guarantee

Suppose a model has been trained, and a further $n$ labelled examples $(X_1, Y_1), \ldots, (X_n, Y_n)$ that the model never saw are set aside for calibration. For a new input $X_{\text{test}}$ with unknown label $Y_{\text{test}}$, conformal prediction produces a set $C(X_{\text{test}})$ such that

1−α≤P(Ytest∈C(Xtest))≤1−α+1n+1,1 - \alpha \le P\big(Y_{\text{test}} \in C(X_{\text{test}})\big) \le 1 - \alpha + \frac{1}{n+1},

where $\alpha$ is the error rate the user picks (for example $\alpha = 0.1$ for 90% coverage).[3] The lower bound is the main guarantee. The upper bound, which says the sets are not much more conservative than they need to be, requires the scores to have no ties, which is handled by random tie-breaking that is usually ignored in practice.[3]

Angelopoulos and Bates call this property marginal coverage, because the probability is averaged over the randomness of both the calibration data and the test point.[3] It does not promise 90% coverage for every individual input, or for every subgroup of inputs; that stronger property is called conditional coverage and is discussed below.[3]

Shafer and Vovk's 2008 tutorial stresses a second sense of validity. Conformal prediction was first designed for an online setting, in which labels are predicted one after another and each label is revealed before the next prediction is made. If the examples are drawn independently from the same distribution, the successive predictions made at confidence level $1 - \epsilon$ are correct $1 - \epsilon$ of the time, even though they all draw on one growing dataset rather than on independent datasets.[2] The tutorial also notes that the prediction regions for different error levels are nested: a region at a lower confidence level is contained in a region at a higher one.[2]

"Emmanuel Candès - A Taste of Conformal Prediction", video from Institut des Hautes Etudes Scientifiques (IHES), published April 26, 2023. Watch on YouTube.

How split conformal prediction works

Angelopoulos and Bates describe the general split conformal recipe in four steps.[3]

  1. Start from a heuristic notion of uncertainty produced by a pretrained model, such as softmax probabilities or an estimated spread.
  2. Define a score function $s(x, y)$, where a larger score means worse agreement between input $x$ and candidate label $y$. Scores of this kind are called nonconformity scores.
  3. Compute the scores $s_1 = s(X_1, Y_1), \ldots, s_n = s(X_n, Y_n)$ on the calibration set, and let $\hat{q}$ be their $\lceil (n+1)(1-\alpha) \rceil / n$ empirical quantile, where $\lceil \cdot \rceil$ is the ceiling function.
  4. For a new input, return the set $C(X_{\text{test}}) = {y : s(X_{\text{test}}, y) \le \hat{q}}$.

The quantile level is slightly above $1 - \alpha$; the $(n+1)$ term is a finite-sample correction that accounts for the test point itself.[3] Angelopoulos and Bates attribute the coverage theorem for this procedure to Vovk, Gammerman and Craig Saunders and give a proof in an appendix of their tutorial.[3] Shafer and Vovk describe the same idea in terms of a nonconformity measure, which "measures how unusual an example looks relative to previous examples".[2]

Classification example

In the introductory example from Angelopoulos and Bates, an image classifier outputs softmax probabilities for $K$ classes, and the score on a calibration example is one minus the softmax probability the model gave to the true class, $s_i = 1 - \hat{f}(X_i)_{Y_i}$.[3] After computing $\hat{q}$, the prediction set for a new image contains every class whose softmax probability is at least $1 - \hat{q}$.[3] Hard or ambiguous images get larger sets, and easy images get small ones, so the size of the set acts as a signal of the model's uncertainty.[3] The authors suggest a calibration set of moderate size, such as 500 examples, in their opening example.[3]

Regression example

For regression, Lei, G'Sell, Rinaldo, Tibshirani and Wasserman (published in the Journal of the American Statistical Association in 2018) give a split conformal algorithm that fits a regression function $\hat{\mu}$ on one half of the data, computes absolute residuals $|Y_i - \hat{\mu}(X_i)|$ on the other half, and returns the interval $[\hat{\mu}(x) - d, \hat{\mu}(x) + d]$, where $d$ is the appropriate order statistic of those residuals.[5] Their theorem shows coverage of at least $1 - \alpha$ for i.i.d. data, and at most $1 - \alpha + 2/(n+2)$ when the residuals have a continuous joint distribution, where $n$ is the total sample size before splitting.[5] This interval has the same width everywhere, which is one of the reasons later methods tried to make intervals adapt to the input.[8]

Choosing the score function

Because the coverage guarantee holds for any score, the score decides whether the sets are informative. Angelopoulos and Bates note that if the scores were random noise, the sets would still achieve valid marginal coverage but would be a random sample of the label space and therefore useless; good scores rank inputs by how wrong the model is likely to be.[3] Some widely cited choices are listed below.

MethodTaskScore ideaSource
Absolute residualRegression$\lvert y - \hat{\mu}(x) \rvert$, giving a fixed-width intervalLei et al. 2018[5]
Conformalized quantile regression (CQR)RegressionDistance from $y$ to the nearer of two fitted quantilesRomano, Patterson and Candès 2019[8]
One minus true-class softmaxClassification$1 - \hat{f}(x)_y$; produces the smallest average setsAngelopoulos and Bates[3]
Adaptive prediction sets (APS)ClassificationCumulative softmax mass of classes ranked above and including $y$Romano, Sesia and Candès 2020[9]
Regularized adaptive prediction sets (RAPS)ClassificationAPS with a penalty on the scores of unlikely classesAngelopoulos, Bates, Malik and Jordan 2021[10]

Expanded article table

Conformalized quantile regression. Romano, Patterson and Candès presented CQR at NeurIPS 2019 as a method "fully adaptive to heteroscedasticity", combining conformal prediction with classical quantile regression.[8] A model is trained to predict the $\alpha/2$ and $1 - \alpha/2$ conditional quantiles, for example by replacing a mean squared error objective with the pinball loss function. The score is $s(x, y) = \max{\hat{t}{\alpha/2}(x) - y,; y - \hat{t}{1-\alpha/2}(x)}$, and the final interval widens or narrows the fitted quantile band by $\hat{q}$: $[\hat{t}{\alpha/2}(x) - \hat{q},; \hat{t}{1-\alpha/2}(x) + \hat{q}]$.[3] In the paper's experiments on standard regression datasets, CQR tended to give shorter intervals than other conformal methods.[8]

Adaptive prediction sets. Angelopoulos and Bates explain that the simple softmax score gives the smallest average set size but tends to undercover hard subgroups and overcover easy ones.[3] APS, developed by Romano, Sesia and Candès, instead sorts the classes from most to least likely and adds up their probabilities until it reaches the true label; this score uses the probabilities of all classes, not only the true one.[3][9] The NeurIPS 2020 paper reports that the method performs favourably in approximate conditional coverage compared with alternatives.[9] RAPS, from Angelopoulos, Bates, Jitendra Malik and Michael I. Jordan (an ICLR 2021 spotlight paper), regularizes the small scores of unlikely classes to make the sets more stable. On ImageNet and ImageNet-V2 with a ResNet-152 and other classifiers, the authors report sets that were often 5 to 10 times smaller than a stand-alone Platt scaling baseline while still meeting the coverage target.[10]

Full, split and cross-conformal prediction

The original form of the method, now called full conformal prediction, does not reserve a separate calibration set. For each candidate label $y$, it refits the model on the training data augmented with the test point $(X_{n+1}, y)$, recomputes the scores, and keeps $y$ in the set if the test point's score is not unusually large compared with the others.[3] The model-fitting algorithm must treat data points symmetrically.[3] Angelopoulos and Bates note that the acceptance rule is equivalent to a permutation test of exchangeability, so the prediction set consists of all labels under which the hypothetical test point looks consistent with the data.[3] The cost is high: with $K$ possible labels the procedure needs on the order of $(n+1) \cdot K$ model fits, and continuous outputs must usually be discretized first.[3]

Split conformal prediction separates fitting from calibration, so its cost is a single model fit.[5] Lei and colleagues trace the idea to the inductive conformal prediction of Harris Papadopoulos, Kostas Proedrou, Vovk and Gammerman, presented at ECML 2002 under the title "Inductive Confidence Machines for Regression".[5][6] The price of the split is statistical efficiency, since part of the data is used only for calibration.[3]

Methods in between reuse all the data for both fitting and calibration at the cost of a moderate number of fits. These include Vovk's cross-conformal predictors and the jackknife+ and CV+ methods of Rina Foygel Barber, Emmanuel Candès, Aaditya Ramdas and Ryan Tibshirani.[3][16] The jackknife+ (published in The Annals of Statistics in 2021) builds intervals from leave-one-out residuals and leave-one-out predictions at the test point; the authors prove coverage guarantees for any algorithm that treats training points symmetrically, and show that such guarantees are not possible for the original jackknife, whose coverage can vanish in some examples.[16] The approach also extends to $K$-fold cross-validation.[16]

Marginal and conditional coverage

The standard guarantee is marginal. Angelopoulos and Bates illustrate the gap with two groups that make up 90% and 10% of a population: a procedure that always covers the first group and never covers the second has 90% marginal coverage but fails the second group completely.[3] Conditional coverage, $P(Y_{\text{test}} \in C(X_{\text{test}}) \mid X_{\text{test}}) \ge 1 - \alpha$, would rule this out, but exact conditional coverage is impossible to achieve in general without assumptions about the distribution.[3][12] Barber, Candès, Ramdas and Tibshirani studied which relaxations of conditional coverage remain achievable in a distribution-free setting.[12]

Several practical steps lie between the two extremes:

  • Group-balanced coverage. If the inputs carry a discrete group label (for example a demographic group or a binned feature), running the calibration separately within each group gives $1 - \alpha$ coverage in every group, a result Angelopoulos and Bates credit to Vovk's 2012 paper on the conditional validity of inductive conformal predictors.[3][11]
  • Class-conditional coverage. In classification, calibrating separately for each true class gives $1 - \alpha$ coverage for every class, so that, in the tutorial's example, a medical classifier covers both the normal class and the cancer class at the target rate.[3]
  • Coverage over classes of shifts. Isaac Gibbs, John Cherian and Candès (posted in 2023 and published in the Journal of the Royal Statistical Society Series B in 2025) reformulated conditional coverage as coverage over a class of covariate shifts and showed how to obtain exact finite-sample coverage over all shifts in a finite-dimensional class, for example over each of a collection of subgroups.[26]

The coverage obtained with one particular calibration set is itself random. Following Vovk's 2012 analysis, the coverage conditional on the calibration data follows a Beta distribution, $\text{Beta}(n + 1 - l, l)$ with $l = \lfloor (n+1)\alpha \rfloor$.[3][11] Based on this, Angelopoulos and Bates found that about 1,000 calibration points usually keep the realized coverage between 0.88 and 0.92 when the target is 0.90, and they suggest that size as a rough guideline.[3]

Beyond exchangeability

Exchangeability fails when the data at deployment differ from the calibration data, a situation covered under distribution shift. Several extensions address it.

Covariate shift. Ryan Tibshirani, Barber, Candès and Ramdas (NeurIPS 2019) showed that a weighted version of conformal prediction remains valid when the distribution of inputs changes but the conditional distribution of the label given the input does not, provided the likelihood ratio between the test and training input distributions is known or can be estimated well from unlabelled test inputs.[13] Calibration points that are more likely under the new distribution receive more weight in the quantile.[3]

Distribution drift and non-symmetric algorithms. In "Conformal prediction beyond exchangeability" (The Annals of Statistics, 2023), Barber, Candès, Ramdas and Tibshirani used weighted quantiles, for example placing more weight on recent observations, to make conformal methods robust to drift, and introduced a randomization technique for model-fitting algorithms that do not treat data points symmetrically.[14] In the tutorial's statement of the result, the loss of coverage is at most twice a weighted sum of the total variation distances between each calibration point's distribution and the test distribution, and the methods keep the usual guarantee when the data are in fact exchangeable.[3][14] The paper demonstrates the methods on electricity and election forecasting data.[14]

Adaptive conformal inference. For data that arrive over time with a distribution that changes in an unknown way, Gibbs and Candès (NeurIPS 2021) proposed adaptive conformal inference (ACI).[15] It treats the miscoverage level used to form the sets as a single parameter $\alpha_t$ and updates it after each observation:

αt+1=αt+γ(α−errt),\alpha_{t+1} = \alpha_t + \gamma(\alpha - \text{err}_t),

where $\text{err}_t$ is 1 if the set at time $t$ missed the true label and 0 otherwise, and $\gamma > 0$ is a step size.[15] The authors prove that, with probability one, the long-run fraction of misses converges to the target $\alpha$ regardless of how the data are generated.[15]

Conformal risk control

Coverage is a statement about one kind of error: the true label falling outside the set. Conformal risk control, introduced by Angelopoulos, Bates, Adam Fisch, Lihua Lei and Tal Schuster in a 2022 preprint and published at ICLR 2024, generalizes split conformal prediction to bound the expected value of any monotone loss function.[17] The procedure includes split conformal prediction and its coverage guarantee as a special case and is tight up to an $O(1/n)$ term.[17] The paper's worked examples from computer vision and natural language processing bound the false negative rate, a graph distance and token-level F1-score.[17] Angelopoulos and Bates's tutorial includes a worked example that controls the false negative rate of a tumour segmentation model.[3]

Use with large language models

Large language models produce free-form text from a very large output space and often generate confident errors, so researchers have adapted conformal methods in several ways. A 2024 survey in the Transactions of the Association for Computational Linguistics by Margarida Campos, António Farinhas, Chrysoula Zerva, Mário Figueiredo and André Martins reviews these techniques and their applications in natural language processing, citing hallucination among the risks that motivate them.[23]

WorkSettingWhat is calibratedSource
Kumar et al. 2023Multiple-choice question answeringSets of answer options from softmax over option letters[18]
KnowNo (Ren et al. 2023)Robot planning with an LLM plannerSets of candidate plans; the robot asks a human when the set has more than one plan[19]
Conformal language modeling (Quach et al. 2024)Open-ended generationA stopping rule for sampling and a rejection rule for candidates[20]
Conformal factuality (Mohri and Hashimoto 2024)Factual claims in long outputsA threshold for removing low-confidence sub-claims[21]
Cherian, Gibbs and Candès 2024Factual claims in long outputsConditionally adaptive claim filtering and a learned scoring function[22]

Expanded article table

Multiple-choice answers. Bhawesh Kumar and colleagues applied split conformal prediction to LLaMA-13B on multiple-choice questions from 16 subjects of the MMLU benchmark, using the softmax over the answer letters as the model's confidence.[18] They found that the size of the conformal sets was closely correlated with accuracy, which could support selective classification, and they examined how the exchangeability assumption holds up for questions from other subjects.[18] The work appeared at an ICML 2023 workshop.[18]

Robots that ask for help. KnowNo, by Allen Ren and colleagues (CoRL 2023), frames each planning step of an LLM-driven robot as multiple-choice question answering: the LLM generates candidate plans labelled A to D plus an option "E) an option not listed here", conformal prediction selects a subset of them, and the robot executes the plan if the set contains a single option and asks a human for help otherwise.[19] The authors report statistical guarantees on task completion while reducing the amount of human help needed, and say the approach works with off-the-shelf LLMs without fine-tuning.[19]

Sets of generated responses. Victor Quach, Fisch, Schuster, Adam Yala, Jae Ho Sohn, Tommi Jaakkola and Regina Barzilay (ICLR 2024) calibrate a stopping rule for sampling outputs from a language model and a rejection rule for discarding low-quality samples, so that the returned set contains at least one acceptable answer with high probability.[20] They also identify individual phrases or sentences that are each correct with statistical guarantees, and test the approach on open-domain question answering, text summarization and radiology report generation.[20]

Factuality guarantees. Christopher Mohri and Tatsunori Hashimoto (ICML 2024) framed the correctness of a language model's output as an uncertainty-set problem in which making the output less specific expands the associated set, which lets conformal prediction act as a back-off procedure: split the output into sub-claims, score them, and remove the least certain ones until the remaining text is correct with the target probability.[21] Using GPT-4 outputs on closed-book question answering and reasoning tasks, they report 80 to 90% correctness guarantees while keeping most of the original output.[21] Cherian, Gibbs and Candès (NeurIPS 2024) pointed out that such guarantees are not conditionally valid, so the reliability of the filter can vary by topic, and that imperfect scores can remove many accurate claims; they proposed conditional conformal methods that issue weaker guarantees where needed and a way to improve the scoring function by differentiating through the conformal procedure, evaluated on biography and medical question answering.[22]

History

Angelopoulos and Bates trace the roots of conformal prediction to Vovk's work as a student of Andrey Kolmogorov at Moscow State University on algorithmically random sequences; in their account, the idea of randomness deficiency later became nonconformity, and the notion of collectives became exchangeability.[3] The framework itself was developed by Vovk, Gammerman, Saunders and Vladimir Vapnik between 1996 and 1999.[3] An early paper, "Learning by Transduction" by Gammerman, Vovk and Vapnik at the UAI 1998 conference, described a modification of the support vector machine that gives a prediction together with a practical measure of the evidence supporting it.[7]

YearDevelopmentSource
1998Gammerman, Vovk and Vapnik, "Learning by Transduction" (UAI)[7]
2002Inductive (split) conformal prediction for regression by Papadopoulos, Proedrou, Vovk and Gammerman (ECML)[3][6]
2003Glenn Shafer coins the term "conformal predictor" on 1 December 2003 while writing the book with Vovk[3]
2005Algorithmic Learning in a Random World by Vovk, Gammerman and Shafer (Springer)[1]
2008Shafer and Vovk, "A Tutorial on Conformal Prediction" (Journal of Machine Learning Research)[2]
2012The COPA symposium (Conformal and Probabilistic Prediction and its Applications) is founded; cross-conformal predictors introduced; Vovk's paper on conditional validity[3][11]
2018Lei et al., "Distribution-Free Predictive Inference for Regression" (JASA)[5]
2019Conformalized quantile regression; weighted conformal prediction under covariate shift (both NeurIPS)[8][13]
2021Adaptive conformal inference (NeurIPS); RAPS for image classifiers (ICLR); jackknife+ (Annals of Statistics)[10][15][16]
2022Second edition of Algorithmic Learning in a Random World[25]
2023Angelopoulos and Bates tutorial published in Foundations and Trends in Machine Learning; "Conformal prediction beyond exchangeability" (Annals of Statistics)[4][14]
2024LLM methods at ICLR, ICML and NeurIPS; Theoretical Foundations of Conformal Prediction posted to arXiv[20][21][22][24]

Expanded article table

According to Angelopoulos and Bates, conformal prediction was popularized in the United States by Jing Lei, Larry Wasserman and colleagues, whose work introduced the method to many statisticians, including Barber, Candès, Ramdas and Tibshirani, who went on to make major contributions.[3] The tutorial by Angelopoulos and Bates, first posted to arXiv in July 2021 with Python code samples and a Jupyter notebook for each, was published as a monograph in Foundations and Trends in Machine Learning in 2023.[3][4] Other applications listed in that tutorial include the Washington Post's estimates of outstanding votes in the 2020 United States presidential election, early clinical experiments, and robotic planning that avoids collisions.[3]

In November 2024, Angelopoulos, Barber and Stephen Bates posted Theoretical Foundations of Conformal Prediction, a book on the proof techniques behind conformal prediction and related methods built on permutation tests and exchangeability; the arXiv record states that it will be published by Cambridge University Press.[24]

Software

Several open-source Python libraries implement conformal methods. The versions below are the latest releases listed on the Python Package Index as of 18 November 2026.

LibraryDescription (from its package listing)Latest versionSource
MAPIE"A scikit-learn-compatible module for estimating prediction intervals"1.6.0[27]
crepes"Conformal classifiers, regressors, and predictive systems"0.9.1[28]
TorchCP"A Python toolbox for conformal prediction on deep learning models"1.2.1[29]
puncc"Predictive UNcertainty Calibration and Conformalization Library"0.9.3[30]

Expanded article table

MAPIE is developed under the scikit-learn-contrib organization on GitHub and works with scikit-learn estimators; TorchCP lists PyTorch among its dependencies.[27][29] Lei and colleagues released an R package, conformalInference, alongside their 2018 paper.[5]

Limitations

  • Marginal, not individual, guarantees. The basic guarantee holds on average over calibration and test data, and can coexist with poor coverage for particular subgroups or inputs.[3]
  • Dependence on exchangeability. When deployment data differ from calibration data, the guarantee no longer holds as stated; the extensions for covariate shift and drift require either a known or estimable shift or accept a bound that weakens with the size of the shift.[3][13][14]
  • Usefulness depends on the model and score. A weak model or a poorly chosen score still produces valid sets, but they can be too large to be informative.[3]
  • Variability with small calibration sets. The realized coverage for one calibration set fluctuates around the target, and the fluctuation shrinks as the calibration set grows.[3]
  • Cost of full conformal prediction. The full method requires refitting the model for each candidate label, which is usually impractical for large models; split conformal prediction avoids this at some loss of statistical efficiency.[3]
  • Defining correctness for generated text. For language models, methods must decide what counts as an acceptable or correct output, and Cherian, Gibbs and Candès note that the resulting guarantees can vary in reliability by topic when they are only marginal.[20][22]

Continue reading

More from See also

More to read

Topics:Machine LearningModel EvaluationStatistics

References

  1. ^1 ^2Vovk, V., Gammerman, A., and Shafer, G. "Algorithmic Learning in a Random World". Springer, 2005. doi.org/...b106715
  2. ^1 ^2 ^3 ^4 ^5Shafer, G., and Vovk, V. "A Tutorial on Conformal Prediction". Journal of Machine Learning Research 9 (2008) 371-421; arXiv:0706.3188. arxiv.org/...0706.3188
  3. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45 ^46 ^47 ^48 ^49Angelopoulos, A. N., and Bates, S. "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification". arXiv:2107.07511, 2021 (version 6, December 2022). arxiv.org/...2107.07511
  4. ^1 ^2Angelopoulos, A. N., and Bates, S. "Conformal Prediction: A Gentle Introduction". Foundations and Trends in Machine Learning 16(4), 494-591, 2023. doi.org/...2200000101
  5. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8Lei, J., G'Sell, M., Rinaldo, A., Tibshirani, R. J., and Wasserman, L. "Distribution-Free Predictive Inference for Regression". Journal of the American Statistical Association 113(523), 1094-1111, 2018. doi.org/...01621459.2017.1307116
  6. ^1 ^2Papadopoulos, H., Proedrou, K., Vovk, V., and Gammerman, A. "Inductive Confidence Machines for Regression". Machine Learning: ECML 2002, Lecture Notes in Computer Science, Springer, 2002, 345-356. doi.org/...3-540-36755-1_29
  7. ^1 ^2Gammerman, A., Vovk, V., and Vapnik, V. "Learning by Transduction". Proceedings of the Fourteenth Conference on Uncertainty in Artificial Intelligence (UAI 1998); arXiv:1301.7375. arxiv.org/...1301.7375
  8. ^1 ^2 ^3 ^4 ^5Romano, Y., Patterson, E., and Candès, E. J. "Conformalized Quantile Regression". Advances in Neural Information Processing Systems 32 (NeurIPS 2019). papers.nips.cc/...b063c431bd1268e9b5e76fb-Abstract
  9. ^1 ^2 ^3Romano, Y., Sesia, M., and Candès, E. J. "Classification with Valid and Adaptive Coverage". Advances in Neural Information Processing Systems 33 (NeurIPS 2020). papers.nips.cc/...5dc81602b7615cd705545f5-Abstract
  10. ^1 ^2 ^3 ^4Angelopoulos, A., Bates, S., Malik, J., and Jordan, M. I. "Uncertainty Sets for Image Classifiers using Conformal Prediction". International Conference on Learning Representations (ICLR 2021); arXiv:2009.14193. arxiv.org/...2009.14193
  11. ^1 ^2 ^3Vovk, V. "Conditional Validity of Inductive Conformal Predictors". Proceedings of the Asian Conference on Machine Learning, PMLR 25:475-490, 2012. proceedings.mlr.press/...vovk12
  12. ^1 ^2Barber, R. F., Candès, E. J., Ramdas, A., and Tibshirani, R. J. "The limits of distribution-free conditional predictive inference". Information and Inference: A Journal of the IMA 10(2), 455-482 (published online 2020). doi.org/...iaaa017
  13. ^1 ^2 ^3Tibshirani, R. J., Foygel Barber, R., Candès, E. J., and Ramdas, A. "Conformal Prediction Under Covariate Shift". Advances in Neural Information Processing Systems 32 (NeurIPS 2019). papers.nips.cc/...2207526da55a679f0332de2-Abstract
  14. ^1 ^2 ^3 ^4 ^5Barber, R. F., Candès, E. J., Ramdas, A., and Tibshirani, R. J. "Conformal prediction beyond exchangeability". The Annals of Statistics 51(2), 816-845, 2023. doi.org/...23-AOS2276
  15. ^1 ^2 ^3 ^4Gibbs, I., and Candès, E. "Adaptive Conformal Inference Under Distribution Shift". Advances in Neural Information Processing Systems 34 (NeurIPS 2021). papers.nips.cc/...945e5acbc865406fc9a2559-Abstract
  16. ^1 ^2 ^3 ^4Barber, R. F., Candès, E. J., Ramdas, A., and Tibshirani, R. J. "Predictive inference with the jackknife+". The Annals of Statistics 49(1), 486-507, 2021. doi.org/...20-AOS1965
  17. ^1 ^2 ^3Angelopoulos, A. N., Bates, S., Fisch, A., Lei, L., and Schuster, T. "Conformal Risk Control". International Conference on Learning Representations (ICLR 2024); arXiv:2208.02814. arxiv.org/...2208.02814
  18. ^1 ^2 ^3 ^4Kumar, B., Lu, C., Gupta, G., Palepu, A., Bellamy, D., Raskar, R., and Beam, A. "Conformal Prediction with Large Language Models for Multi-Choice Question Answering". ICML 2023 Neural Conversational AI (TEACH) workshop; arXiv:2305.18404. arxiv.org/...2305.18404
  19. ^1 ^2 ^3Ren, A. Z., Dixit, A., Bodrova, A., Singh, S., Tu, S., Brown, N., Xu, P., Takayama, L., Xia, F., Varley, J., Xu, Z., Sadigh, D., Zeng, A., and Majumdar, A. "Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners". Conference on Robot Learning (CoRL 2023); arXiv:2307.01928. arxiv.org/...2307.01928
  20. ^1 ^2 ^3 ^4 ^5Quach, V., Fisch, A., Schuster, T., Yala, A., Sohn, J. H., Jaakkola, T. S., and Barzilay, R. "Conformal Language Modeling". International Conference on Learning Representations (ICLR 2024); arXiv:2306.10193. arxiv.org/...2306.10193
  21. ^1 ^2 ^3 ^4Mohri, C., and Hashimoto, T. "Language Models with Conformal Factuality Guarantees". Proceedings of the 41st International Conference on Machine Learning, PMLR 235:36029-36047, 2024; arXiv:2402.10978. arxiv.org/...2402.10978
  22. ^1 ^2 ^3 ^4Cherian, J. J., Gibbs, I., and Candès, E. J. "Large language model validity via enhanced conformal prediction methods". Advances in Neural Information Processing Systems 37 (NeurIPS 2024). papers.nips.cc/...90dd1ad21526-Abstract-Conference
  23. ^1 ^2Campos, M. M., Farinhas, A., Zerva, C., Figueiredo, M. A. T., and Martins, A. F. T. "Conformal Prediction for Natural Language Processing: A Survey". Transactions of the Association for Computational Linguistics 12, 1497-1516, 2024. doi.org/...tacl_a_00715
  24. ^1 ^2Angelopoulos, A. N., Barber, R. F., and Bates, S. "Theoretical Foundations of Conformal Prediction". arXiv:2411.11824, 2024. arxiv.org/...2411.11824
  25. ^Vovk, V., Gammerman, A., and Shafer, G. "Algorithmic Learning in a Random World" (second edition). Springer, 2022. doi.org/...978-3-031-06649-8
  26. ^Gibbs, I., Cherian, J. J., and Candès, E. J. "Conformal prediction with conditional guarantees". Journal of the Royal Statistical Society Series B: Statistical Methodology 87(4), 1100-1126, 2025; arXiv:2305.12616. doi.org/...qkaf008
  27. ^1 ^2Python Package Index. "MAPIE". PyPI, accessed 18 November 2026. pypi.org/...MAPIE
  28. ^Python Package Index. "crepes". PyPI, accessed 18 November 2026. pypi.org/...crepes
  29. ^1 ^2Python Package Index. "torchcp". PyPI, accessed 18 November 2026. pypi.org/...torchcp
  30. ^Python Package Index. "puncc". PyPI, accessed 18 November 2026. pypi.org/...puncc

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

2 revisions · v2 · 4,743 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check before publication on November 18, 2026: 118 claims checked against 34 cited and independent sources; 0 material and 3 minor corrections applied before publication.

Cite this page: AI Wiki. "Conformal prediction." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/conformal_prediction

Edit article