Bayesian statistics

Bayesian statistics is a statistical paradigm in which probability expresses a degree of belief that is updated as evidence arrives, using Bayes' theorem. The central rule is that the posterior is proportional to the likelihood times the prior: the updated belief about a parameter after seeing data, $P(\theta \mid D)$, is proportional to the probability of the data under that parameter, $P(D \mid \theta)$, multiplied by the prior belief $P(\theta)$ held before the data arrived ^[1][2]. As a standard primer puts it, "Bayesian statistics is an approach to data analysis based on Bayes' theorem, where available knowledge about parameters in a statistical model is updated with the information in observed data" ^[1].

This interpretation contrasts with frequentist statistics, where probability refers to the long-run frequency of an event in repeated experiments and parameters are treated as fixed but unknown constants with no prior distribution ^[3]. In the Bayesian framework, parameters are themselves random variables with probability distributions, and the central object of inference is the posterior distribution over those parameters given the observed data ^[1][3].

The approach is named for the Reverend Thomas Bayes, whose essay communicated posthumously in 1763 laid the mathematical foundations, but it took roughly two centuries, several philosophical revivals, and a computational revolution before Bayesian methods became practical at the scale modern science demands ^[4][6]. Today the framework is used in clinical trial design, machine learning, cosmology, election forecasting, and large parts of cognitive science, and it underlies a thriving ecosystem of probabilistic programming languages such as Stan, PyMC, and NumPyro ^[1][9][14]. It is also central to modern artificial intelligence: Bayesian inference, Bayesian networks, Bayesian optimization for hyperparameter tuning, Markov chain Monte Carlo and variational inference for approximate posteriors, and Bayesian deep learning for uncertainty quantification all build directly on the posterior-proportional-to-likelihood-times-prior rule ^[1].

What is the fundamental equation of Bayesian statistics?

All of Bayesian statistics rests on a single rearrangement of the definition of conditional probability. Given a hypothesis or parameter $H$ and observed data $D$:

$P(H \mid D) = \frac{P(D \mid H)\,P(H)}{P(D)}$

A more compact and frequently quoted form is:

$\text{Posterior} \propto \text{Likelihood} \times \text{Prior}$

This proportionality holds because the denominator $P(D)$ does not depend on $H$, so it acts only as a normalising constant ^[1][2]. The four ingredients each have a precise role:

Scroll sideways for more →

In most realistic problems the integral defining $P(D)$ has no closed form, so a great deal of Bayesian computation is devoted either to evaluating that integral, sidestepping it, or sampling from the unnormalised posterior $P(D \mid H),P(H)$ directly ^[9][22].

What are the core concepts of Bayesian statistics?

Bayesian statistics has its own vocabulary, much of it shared with Bayesian inference and probabilistic modelling more broadly. The prior, likelihood, and posterior form the central framework: the prior encodes belief before the data, the likelihood links parameters to data, and the posterior is the result of combining the two through Bayes' theorem ^[1].

Scroll sideways for more →

When was Bayesian statistics developed?

The history of Bayesian statistics is unusually contested for a mathematical subject; the framework spent more than a century in the wilderness before re-emerging as a dominant approach.

The Reverend Thomas Bayes, an English Presbyterian minister, formulated his theorem some time in the late 1740s. He never published it. After his death in 1761 his friend Richard Price edited the manuscript and read it before the Royal Society in December 1763 as "An Essay towards solving a Problem in the Doctrine of Chances" ^[4]. The essay tackled the inverse probability problem of inferring an unknown chance from observed successes, and it included Price's own introduction explaining the philosophical motivation ^[4].

Pierre-Simon Laplace developed the same ideas independently and far more systematically. His 1812 "Théorie analytique des probabilités" made what we now call Bayesian inference a workhorse of nineteenth century science, applied to problems in astronomy, demography, and the reliability of legal testimony ^[6]. For Laplace, probability was the natural language of partial knowledge, and assigning prior probabilities was simply a matter of insufficient reason ^[6].

The twentieth century saw a sharp turn against this view. Ronald Fisher, Jerzy Neyman, and Egon Pearson built the frequentist edifice between roughly 1920 and 1940: maximum likelihood, significance tests, p-values, confidence intervals, and the Neyman-Pearson framework for hypothesis testing ^[3]. Their work pushed Bayesian methods to the margins of academic statistics, in part because the priors looked subjective and arbitrary, and in part because the resulting integrals were generally impossible to compute.

A quiet revival began in the 1930s and 1940s. Bruno de Finetti's 1937 paper "La prévision: ses lois logiques, ses sources subjectives" gave subjective probability a rigorous behavioural foundation, deriving the rules of probability from coherent betting odds and proving the famous representation theorem for exchangeable sequences ^[5]. Leonard Savage's 1954 "The Foundations of Statistics" combined de Finetti's subjective probability with the von Neumann-Morgenstern theory of utility to produce the canonical axiomatisation of Bayesian decision theory ^[7]. Edwin Jaynes argued through the 1950s and 1960s that probability theory is simply extended logic, a view eventually compiled in his posthumous 2003 book "Probability Theory: The Logic of Science", which presents "probability theory as extended logic" applied to reasoning under uncertainty ^[8].

The practical breakthrough was computational. The Metropolis algorithm appeared in a 1953 physics paper by Metropolis, Rosenbluth, Rosenbluth, Teller, and Teller, and was generalised to non-symmetric proposals by W. K. Hastings in 1970 ^[10][11]. Stuart and Donald Geman introduced the Gibbs sampler in 1984 for image restoration ^[12]. Once Alan Gelfand and Adrian Smith showed in 1990 how Gibbs sampling could be applied to mainstream Bayesian problems, Markov chain Monte Carlo (MCMC) became the engine that let Bayesian inference handle realistic models ^[13]. The 2014 paper by Matthew Hoffman and Andrew Gelman introduced the No-U-Turn Sampler (NUTS), an adaptive form of Hamiltonian Monte Carlo that powers modern Stan and freed users from manually tuning the number of leapfrog steps ^[22].

The last fifteen years brought probabilistic programming. Stan, PyMC, Pyro, NumPyro, and TensorFlow Probability turned model specification into something close to scripting, and they made Bayesian inference accessible outside the small community of statisticians willing to write their own samplers ^[14][15][16].

How is Bayesian statistics different from frequentist statistics?

The two paradigms answer different questions and use the same data in different ways. The frequentist treats probability as a long-run frequency and parameters as fixed unknown constants, applying probability only to data arising from repeated sampling; the Bayesian treats probability as a degree of belief and parameters as random variables, and forms a posterior by combining a prior with the likelihood ^[1][3]. The differences are real but often less stark in practice than the philosophical debates suggest.

Scroll sideways for more →

A classic example: a frequentist 95% confidence interval is constructed so that, across many hypothetical repetitions of the experiment, 95% of intervals computed this way will contain the true parameter ^[3]. A 95% Bayesian credible interval is one in which, given this particular data set and prior, the posterior probability that the parameter lies in the interval is 95% ^[1]. The Bayesian statement is the one most people think a confidence interval means.

What are the most common Bayesian models?

Most applied Bayesian work uses a small set of recurring building blocks. Many of the simplest models exploit conjugacy, where prior and likelihood are mathematically matched so the posterior has a known form.

Scroll sideways for more →

How is a posterior actually computed?

The computational story is the reason Bayesian methods went from a niche philosophical position in the 1970s to a default tool in the 2020s. There is no single way to compute a posterior; choice of method depends on the model size, structure, and how much accuracy you need. The two dominant families for hard problems are Markov chain Monte Carlo, which draws samples from the posterior, and variational inference, which turns posterior approximation into an optimisation problem by minimising the Kullback-Leibler divergence from the true posterior to a simpler family ^[18][22].

Scroll sideways for more →

MCMC convergence diagnostics are a routine part of Bayesian workflow. Effective sample size, the Gelman-Rubin $\hat R$ statistic, trace plots, and posterior predictive checks are standard tools, and good Bayesian software reports them by default ^[9].

What software is used for Bayesian inference?

Probabilistic programming languages let users describe a generative model in code and leave the inference engine to handle posterior computation.

Scroll sideways for more →

These tools share a common pattern: write the model once in a high-level language, get gradients automatically through differentiable programming, and run any of several inference algorithms without rewriting the model. That separation between model and inference is what made probabilistic programming possible.

How is Bayesian statistics used in machine learning?

Many machine learning techniques have a Bayesian reading, and several were originally derived from a Bayesian starting point. Three families are especially central to modern AI: Bayesian optimization, which uses a Gaussian process surrogate and an acquisition function to tune expensive black-box functions such as model hyperparameters in far fewer evaluations than grid or random search ^[20]; approximate-posterior methods (MCMC and variational inference) that make inference tractable in large models ^[18][22]; and Bayesian deep learning, which places distributions over neural network weights to quantify predictive uncertainty ^[21].

Scroll sideways for more →

Where is Bayesian statistics used in the real world?

Bayesian statistics is no longer a niche tool. It is mainstream across science, industry, and policy.

• Clinical trials and drug development. The U.S. Food and Drug Administration has issued guidance for Bayesian designs for medical devices, and adaptive Bayesian designs are used in oncology and rare-disease trials where data are scarce and prior information is valuable ^[24].
• A/B testing in technology companies. Bayesian-flavoured tests give a continuously updated probability that one variant is better than another, avoiding many of the multiple-testing pitfalls of repeated frequentist peeking.
• Genetics. BayesC, BayesR, and BSLMM are widely used Bayesian methods for genomic prediction and genome-wide association studies, where many small effects need to be regularised.
• Search and ranking. Bayesian linear regression and Thompson sampling are standard tools for click-through rate estimation and online ranking.
• Cancer staging and medical diagnosis, where posterior probabilities of disease given test results are the natural output of a diagnostic model.
• Financial risk modelling, including Value at Risk under parameter uncertainty and Bayesian forecasting of macroeconomic time series.
• Astrophysics. The LIGO and Virgo collaborations use Bayesian inference for gravitational wave parameter estimation; exoplanet detection routinely uses nested sampling and Bayesian model comparison.
• Marketing mix modelling, where hierarchical Bayesian regressions estimate the contribution of each channel under realistic priors and constraints.
• Election forecasting. Andrew Gelman's models for The Economist and Nate Silver's models at FiveThirtyEight blend polling, fundamentals, and prior information using hierarchical Bayesian regression.
• Reliability engineering, where Weibull models with Bayesian priors are used for failure-time analysis under sparse data.
• Forensic science, where likelihood ratios and Bayesian reasoning are the formal language of evidence interpretation in DNA analysis.

What are the strengths of Bayesian methods?

Bayesian methods have a number of properties that make them attractive in science and industry alike.

• Uncertainty is handled in a principled way. The posterior is itself a probability distribution, and predictions automatically account for parameter uncertainty ^[1].
• Updating with new data is coherent. The posterior after one batch of data is the prior for the next, which makes streaming and sequential analyses natural ^[1].
• Hierarchical models give partial pooling that interpolates smoothly between treating groups identically and treating them independently. This is one of the most consequential ideas in modern applied statistics ^[9].
• Decision theory plugs in directly: combine the posterior with a loss function and minimise expected loss to get an optimal decision ^[7].
• Posterior predictive checks let you criticise a model by simulating data from the fitted model and comparing it to the observed data ^[9].
• The framework does not rely on asymptotic approximations or large-sample assumptions. Inference for small samples is no different in principle from inference for large samples.

What are the weaknesses of Bayesian methods?

The approach also has real costs.

• Computation is expensive. Even with modern MCMC and variational inference, fitting a complex Bayesian model can take orders of magnitude longer than a comparable frequentist analysis ^[22].
• Prior selection can be controversial. Two analysts with different priors can reach different conclusions from the same data, and writing down a prior for an unfamiliar parameter is genuinely hard ^[1].
• Communication is harder. Many practitioners and most regulators are trained in frequentist language, and credible intervals, posterior probabilities, and Bayes factors take more explanation than p-values.
• MCMC convergence diagnostics matter. A poorly mixed chain can give misleading answers, and detecting that requires care ^[9].
• The framework assumes parameters have a meaningful probability distribution, which some philosophers and statisticians regard as a strong metaphysical commitment.

Why has Bayesian statistics resurged?

Bayesian methods have moved from the periphery to the centre of several fields. In cognitive science the Bayesian brain hypothesis treats perception and learning as approximate Bayesian inference. In industry, hierarchical Bayesian models are standard for marketing analytics, recommendation systems, and reliability engineering. In machine learning, Bayesian deep learning is an active research area motivated by uncertainty quantification, model calibration, and AI safety ^[21]. In medical statistics the FDA has accepted Bayesian designs for confirmatory trials in selected settings, and adaptive Bayesian designs are routine in oncology platform trials ^[24]. In the debate around the reproducibility crisis, prominent statisticians have argued that Bayesian model comparison and posterior reporting should partly replace null hypothesis significance testing.

None of this means the frequentist tradition has been displaced. Most introductory statistics courses still teach the Neyman-Pearson framework, and most published p-values come from frequentist procedures. What has changed is that Bayesian methods are now a serious alternative rather than a fringe view, with mature software and large user communities.

References

1. van de Schoot, R., Depaoli, S., King, R., Kramer, B., Märtens, K., Tadesse, M. G., Vannucci, M., Gelman, A., Veen, D., Willemsen, J., & Yau, C. (2021). "Bayesian statistics and modelling." *Nature Reviews Methods Primers*, 1, Article 3. DOI: 10.1038/s43586-020-00003-0. ↩
2. Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). *Bayesian Data Analysis*, 3rd edition. Chapman & Hall / CRC Press. ↩
3. Wasserman, L. (2004). *All of Statistics: A Concise Course in Statistical Inference*. Springer. (See the comparison of frequentist and Bayesian inference.) ↩
4. Bayes, T. (1763). "An Essay towards solving a Problem in the Doctrine of Chances." *Philosophical Transactions of the Royal Society of London*, 53, 370-418. Communicated posthumously by Richard Price. ↩
5. de Finetti, B. (1937). "La prévision: ses lois logiques, ses sources subjectives." *Annales de l'Institut Henri Poincaré*, 7, 1-68. English translation: "Foresight: Its Logical Laws, Its Subjective Sources." ↩
6. Laplace, P.-S. (1812). *Théorie analytique des probabilités*. Courcier. ↩
7. Savage, L. J. (1954). *The Foundations of Statistics*. John Wiley & Sons. ↩
8. Jaynes, E. T. (2003). *Probability Theory: The Logic of Science*. Cambridge University Press. Edited by G. Larry Bretthorst. ↩
9. Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). *Bayesian Data Analysis*, 3rd edition. Chapman & Hall / CRC Press. (Convergence diagnostics, hierarchical models, and posterior predictive checks.) ↩
10. Metropolis, N., Rosenbluth, A. W., Rosenbluth, M. N., Teller, A. H., & Teller, E. (1953). "Equation of State Calculations by Fast Computing Machines." *Journal of Chemical Physics*, 21(6), 1087-1092. ↩
11. Hastings, W. K. (1970). "Monte Carlo Sampling Methods Using Markov Chains and Their Applications." *Biometrika*, 57(1), 97-109. ↩
12. Geman, S., & Geman, D. (1984). "Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images." *IEEE Transactions on Pattern Analysis and Machine Intelligence*, PAMI-6(6), 721-741. ↩
13. Gelfand, A. E., & Smith, A. F. M. (1990). "Sampling-Based Approaches to Calculating Marginal Densities." *Journal of the American Statistical Association*, 85(410), 398-409. ↩
14. Carpenter, B., Gelman, A., Hoffman, M. D., Lee, D., Goodrich, B., Betancourt, M., Brubaker, M., Guo, J., Li, P., & Riddell, A. (2017). "Stan: A Probabilistic Programming Language." *Journal of Statistical Software*, 76(1). ↩
15. Salvatier, J., Wiecki, T. V., & Fonnesbeck, C. (2016). "Probabilistic Programming in Python Using PyMC3." *PeerJ Computer Science*, 2, e55. ↩
16. Bingham, E., Chen, J. P., Jankowiak, M., Obermeyer, F., Pradhan, N., Karaletsos, T., Singh, R., Szerlip, P., Horsfall, P., & Goodman, N. D. (2018). "Pyro: Deep Universal Probabilistic Programming." *Journal of Machine Learning Research*, 20. ↩
17. Blei, D. M., Ng, A. Y., & Jordan, M. I. (2003). "Latent Dirichlet Allocation." *Journal of Machine Learning Research*, 3, 993-1022. ↩
18. Blei, D. M., Kucukelbir, A., & McAuliffe, J. D. (2017). "Variational Inference: A Review for Statisticians." *Journal of the American Statistical Association*, 112(518), 859-877. ↩
19. Phan, D., Pradhan, N., & Jankowiak, M. (2019). "Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro." arXiv:1912.11554. ↩
20. Snoek, J., Larochelle, H., & Adams, R. P. (2012). "Practical Bayesian Optimization of Machine Learning Algorithms." *Advances in Neural Information Processing Systems*, 25. ↩
21. Gal, Y., & Ghahramani, Z. (2016). "Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning." *Proceedings of the 33rd International Conference on Machine Learning*, 1050-1059. ↩
22. Hoffman, M. D., & Gelman, A. (2014). "The No-U-Turn Sampler: Adaptively Setting Path Lengths in Hamiltonian Monte Carlo." *Journal of Machine Learning Research*, 15, 1593-1623. ↩
23. Maddox, W. J., Izmailov, P., Garipov, T., Vetrov, D. P., & Wilson, A. G. (2019). "A Simple Baseline for Bayesian Uncertainty in Deep Learning." *Advances in Neural Information Processing Systems*, 32. ↩
24. U.S. Food and Drug Administration (2010). "Guidance for the Use of Bayesian Statistics in Medical Device Clinical Trials." Center for Devices and Radiological Health. ↩