Citation and evidence

Probability distribution

20 min full readUpdated 22 references

This article's verification

Report a problem with this article

Edit article
More

Use this article

Raw MarkdownExplore connections

Improve this page

Revision historyDiscuss this article

Topics

Machine LearningMathematicsStatistics

Cite this article

A probability distribution is a description of how likely a random variable, or a set of random variables, is to take on each of its possible values [1]. A distribution over a discrete variable is usually written as a probability mass function (PMF), and a distribution over a continuous variable as a probability density function (PDF); both can be summarized by a cumulative distribution function (CDF) [3]. Machine learning relies on them throughout. Training data are modeled as samples from an unknown data-generating distribution, most neural networks are trained by maximum likelihood so that the distribution they define matches the data [4][5], and generative models try to represent and sample from complex distributions over data such as images [15][18].

Graph with red vertical bars of height 0.2, 0.5 and 0.3 above the values 1, 3 and 7 on a number line
The probability mass function of a discrete distribution that assigns probabilities 0.2, 0.5 and 0.3 to the values 1, 3 and 7. Oleg Alexandrov, via Wikimedia Commons, Public domain.

Definition

Random variables and the axioms of probability

A random variable is a variable that can take on different values randomly. On its own it only lists the states that are possible; it has to be paired with a probability distribution that says how likely each state is [1]. A discrete random variable has a finite or countably infinite number of states, which need not be numbers at all (they can be named categories), while a continuous random variable takes real values [1].

The standard mathematical foundation is the axiomatization in A. N. Kolmogorov's Foundations of the Theory of Probability (1933). A probability function assigns each event $A$ a number with $P(A) \ge 0$, gives the whole outcome space $\Omega$ probability $P(\Omega) = 1$, and is additive over disjoint events; Kolmogorov strengthened additivity from finite to countable unions, a choice that philosophers of probability still debate [2].

Discrete distributions: the probability mass function

For a discrete variable $x$, a PMF $P$ maps each state to the probability that the variable takes that state. To be a valid PMF, $P$ must be defined on every possible state, must satisfy $0 \le P(x) \le 1$ for each state, and must be normalized so that $\sum_x P(x) = 1$ [1]. A uniform distribution over $k$ states, for example, sets $P(x = x_i) = 1/k$ for every $i$, which sums to $k/k = 1$ [1]. A distribution over several variables at once, $P(x, y)$, is called a joint probability distribution [1].

Continuous distributions: the probability density function

For a continuous variable, a PDF $p$ must be non-negative everywhere and must integrate to 1, but unlike a PMF it is not required to stay below 1 [1]. The density at a point is not a probability. Instead, the probability of landing in a tiny region of volume $\delta x$ is approximately $p(x),\delta x$, and the probability that $x$ falls in an interval $[a, b]$ is the integral of $p(x)$ over that interval [1]. The uniform density on $[a, b]$, for instance, equals $1/(b - a)$ inside the interval and 0 outside it [1].

Cumulative distribution function and quantiles

The cumulative distribution function gives the probability that a variable is less than or equal to a value: $F(x) = \Pr[X \le x]$. For a continuous distribution it is the integral of the density from $-\infty$ to $x$, and for a discrete distribution it is the sum of the PMF over all values up to $x$ [3]. Several related functions are defined from it [3]:

FunctionDefinition
Cumulative distribution function$F(x) = \Pr[X \le x]$
Percent point (quantile) functionThe inverse of the CDF
Survival function$S(x) = \Pr[X > x] = 1 - F(x)$
Hazard function$h(x) = f(x) / S(x)$, the density divided by the survival function

Expanded article table

"The Main Ideas behind Probability Distributions", video from StatQuest with Josh Starmer, published April 17, 2017. Watch on YouTube.

Joint, marginal and conditional distributions

Most models in machine learning involve many variables, so a handful of rules for moving between joint and partial descriptions are used constantly [1]:

  • Marginalization (sum rule). The distribution over a subset of variables is found by summing out the others, $P(x) = \sum_y P(x, y)$, or by integrating them out in the continuous case. The name comes from writing the joint table on paper and recording row sums in the margin [1].
  • Conditional probability. $P(y \mid x) = P(y, x) / P(x)$, defined only when $P(x) > 0$ [1]. Goodfellow, Bengio and Courville stress that conditioning is not the same as intervening: the probability that a person is from Germany given that they speak German is high, but teaching someone German does not change where they come from. Questions about actions belong to causal modeling [1].
  • Chain rule (product rule). Any joint distribution factorizes into one-variable conditionals, $P(x^{(1)}, \ldots, x^{(n)}) = P(x^{(1)}) \prod_{i=2}^{n} P(x^{(i)} \mid x^{(1)}, \ldots, x^{(i-1)})$ [1].
  • Independence. Two variables are independent when their joint distribution is the product of their marginals, and conditionally independent given $z$ when that factorization holds for every value of $z$ [1].
  • Bayes' rule. $P(x \mid y) = P(x) P(y \mid x) / P(y)$, where the denominator can be computed as $\sum_x P(y \mid x) P(x)$. The rule is named after Thomas Bayes, who discovered a special case; Pierre-Simon Laplace independently found the general version [1]. The same identity is known as Bayes' theorem, and it is the basis of Bayesian inference.

Factorizing a joint distribution into small factors is the idea behind structured probabilistic models, also called graphical models, which come in directed and undirected forms. Such factorizations can greatly reduce the number of parameters needed to describe a distribution [1].

Expectation, variance and covariance

The expectation of a function $f(x)$ under a distribution is its average value when $x$ is drawn from that distribution: $\mathbb{E}_{x \sim P}[f(x)] = \sum_x P(x) f(x)$ for discrete variables and $\int p(x) f(x),dx$ for continuous ones. Expectation is linear [1]. The variance, $\mathrm{Var}(f(x)) = \mathbb{E}[(f(x) - \mathbb{E}[f(x)])^2]$, measures how much values spread around their mean, and its square root is the standard deviation [1]. Covariance measures how two quantities vary together. Independent variables have zero covariance, but zero covariance does not imply independence, because covariance only detects linear dependence: if $x$ is uniform on $[-1, 1]$ and $y = sx$ for a random sign $s$, then $x$ and $y$ are dependent yet have zero covariance [1].

What probabilities mean

The same rules serve two readings of probability. In the frequentist reading, a probability is the proportion of times an outcome would occur if an experiment were repeated many times; in the Bayesian reading, it is a degree of belief, as when a doctor says a patient has a 40 percent chance of having the flu [1][2]. The Stanford Encyclopedia of Philosophy describes frequency interpretations as identifying probability with relative frequency, and subjective (Bayesian or personalist) interpretations as treating probabilities as degrees of confidence or partial belief [2]. Goodfellow and colleagues note that a small set of common-sense requirements forces degrees of belief to obey the same axioms as frequencies, so machine learning uses one set of formulas for both [1].

They also list three sources of the uncertainty that distributions are used to model: inherent randomness in the system, incomplete observability (a deterministic system can look random when some of its variables are hidden), and incomplete modeling (a model that discards information becomes uncertain about what it discarded) [1].

Common distributions

A few families appear again and again in machine learning. The table summarizes the ones covered here; the parameters given are the standard ones in the cited references.

DistributionTypeSupportPMF or PDFNotes
BernoulliDiscrete${0, 1}$$\phi^x (1 - \phi)^{1-x}$Binary labels; output of a sigmoid unit [1][5]
Categorical (multinoulli)Discrete$k$ categories$P(x = i) = p_i$Class labels; output of a softmax [1][5]
BinomialDiscrete$0, 1, \ldots, n$$\binom{n}{x} p^x (1-p)^{n-x}$Counts of successes in $n$ trials [6]
PoissonDiscrete$0, 1, 2, \ldots$$e^{-\lambda} \lambda^x / x!$Event counts in an interval [7]
UniformContinuous$[a, b]$$1/(b - a)$Zero density outside $[a, b]$ [1]
Gaussian (normal)Continuous$\mathbb{R}$$\sqrt{\tfrac{1}{2\pi\sigma^2}} \exp\left(-\tfrac{(x-\mu)^2}{2\sigma^2}\right)$Default for real-valued noise and targets [1][5]
ExponentialContinuous$x \ge 0$$\lambda e^{-\lambda x}$A sharp peak at zero [1]
LaplaceContinuous$\mathbb{R}$$\tfrac{1}{2\gamma} \exp\left(-\tfrac{\lvert x - \mu \rvert}{\gamma}\right)$A sharp peak at an arbitrary point $\mu$ [1]
BetaContinuous$[0, 1]$$x^{p-1}(1-x)^{q-1} / B(p, q)$Mean $p/(p+q)$ [8]
Dirac / empiricalGeneralizedData points$\tfrac{1}{m} \sum_i \delta(x - x^{(i)})$The training set viewed as a distribution [1]

Expanded article table

Bernoulli, categorical, binomial and Poisson

The Bernoulli distribution covers a single binary variable and has one parameter $\phi \in [0, 1]$, the probability that the variable equals 1; its mean is $\phi$ and its variance $\phi(1 - \phi)$ [1]. The categorical distribution, which Goodfellow and colleagues call the multinoulli, covers one discrete variable with $k$ states. The word "multinoulli" was coined by Gustavo Lacerda and popularized by Murphy (2012); many texts call this distribution "multinomial", although the multinomial properly describes the counts of each category after $n$ draws, with the categorical being the $n = 1$ case [1]. Because their domains can be enumerated, Bernoulli and categorical distributions can represent any distribution over those domains [1].

The binomial distribution gives the probability of $x$ successes in $n$ independent trials that each succeed with a fixed probability $p$. Its mean is $np$, its maximum likelihood estimate of $p$ is $x/n$, and the NIST/SEMATECH e-Handbook of Statistical Methods calls it "probably the most commonly used discrete distribution" [6]. The Poisson distribution models the number of events in a given time interval; its single parameter $\lambda$ is both the mean count and the variance (the standard deviation is $\sqrt{\lambda}$), and its maximum likelihood estimate is the sample mean [7].

Gaussian distribution

The normal distribution, or Gaussian, is the most commonly used distribution over real numbers. It is parameterized by a mean $\mu$, which is also the location of its peak, and a standard deviation $\sigma$; when the density must be evaluated many times, it is often parameterized by the precision $\beta = 1/\sigma^2$ instead [1]. Goodfellow and colleagues give two reasons why it is a sensible default when nothing else is known. First, the central limit theorem shows that the sum of many independent random variables is approximately normal, so many complicated systems can be modeled as Gaussian noise. Second, among all distributions with the same variance, the normal distribution carries the most uncertainty, so it adds the least prior knowledge to a model [1].

The multivariate normal distribution generalizes this to vectors, with a mean vector $\boldsymbol{\mu}$ and a positive definite covariance matrix $\boldsymbol{\Sigma}$. In practice the covariance is often restricted to a diagonal matrix, or to a scalar times the identity (an isotropic Gaussian), which is cheaper to evaluate [1].

Mixtures and latent variables

Richer distributions can be built by mixing simple ones. A mixture distribution first draws a component identity $c$ from a categorical distribution and then draws $x$ from that component, so $P(x) = \sum_i P(c = i) P(x \mid c = i)$ [1]. The component identity is an example of a latent variable, a random variable that is never observed directly. In a Gaussian mixture model each component is a Gaussian with its own mean and covariance, and such a model is a universal approximator of densities: with enough components it can approximate any smooth density to any nonzero error [1]. The empirical distribution of a real-valued dataset is itself a mixture, with one Dirac delta component at each training example; it is also the distribution that maximizes the likelihood of the training data [1].

Measuring how distributions differ

Information theory supplies most of the tools used to compare distributions. The self-information of an outcome is $-\log P(x)$, and the Shannon entropy $H(P) = -\mathbb{E}_{x \sim P}[\log P(x)]$ is the expected information of a draw; nearly deterministic distributions have low entropy and near-uniform ones high entropy [1].

The Kullback-Leibler divergence between two distributions over the same variable is

DKL(P ∥ Q)=Ex∼P[log⁡P(x)−log⁡Q(x)].D_{\mathrm{KL}}(P \,\|\, Q) = \mathbb{E}_{x \sim P}\left[\log P(x) - \log Q(x)\right].

It is non-negative and is zero only when the two distributions are equal (almost everywhere, for continuous variables), but it is not symmetric, so it is not a true distance [1]. The direction matters in practice. When a single Gaussian $q$ is fitted to a two-peaked distribution $p$, minimizing $D_{\mathrm{KL}}(p ,|, q)$ tends to spread $q$ over both peaks, while minimizing $D_{\mathrm{KL}}(q ,|, p)$ tends to lock onto one peak if the peaks are well separated [1].

The cross-entropy $H(P, Q) = -\mathbb{E}{x \sim P}[\log Q(x)]$ equals $H(P) + D{\mathrm{KL}}(P ,|, Q)$. Because $H(P)$ does not depend on $Q$, minimizing cross-entropy with respect to $Q$ is the same as minimizing the KL divergence [1].

Two other measures are prominent in generative modeling. The Jensen-Shannon divergence compares each distribution with their average mixture; it is symmetric and always defined [9]. The Earth-Mover or Wasserstein-1 distance is the smallest expected distance needed to move probability mass from one distribution to the other, taken over all joint distributions whose marginals are the two distributions being compared [9]. It is the basis of the Wasserstein loss.

Probability distributions in machine learning

The data-generating distribution

Statistical learning theory treats datasets as samples from a probability distribution. Under the usual i.i.d. assumptions, examples are independent of one another and the training and test sets are drawn from the same distribution, called the data-generating distribution $p_{\text{data}}$; this is what makes it possible to relate training error to test error [4]. The article on independently and identically distributed data covers the assumption itself, and distribution shift covers what happens when deployment data stop following the training distribution.

Maximum likelihood and loss functions

Maximum likelihood estimation chooses parameters $\theta$ of a model distribution $p_{\text{model}}(x; \theta)$ that maximize $\sum_i \log p_{\text{model}}(x^{(i)}; \theta)$ over the training examples. Dividing by the number of examples turns this into an expectation under the empirical distribution $\hat{p}{\text{data}}$, and maximizing it is equivalent to minimizing $D{\mathrm{KL}}(\hat{p}{\text{data}} ,|, p{\text{model}})$, or equivalently the cross-entropy between the empirical and model distributions [4]. Goodfellow and colleagues argue that calling only the Bernoulli or softmax negative log-likelihood "cross-entropy" is a misnomer, since any negative log-likelihood is a cross-entropy between the empirical distribution and the model [4].

This view explains why common loss functions look the way they do. Most modern neural networks are trained by maximum likelihood, so the cost is the negative log-likelihood $-\mathbb{E}\log p_{\text{model}}(y \mid x)$ [5]. The choice of output distribution then fixes the loss:

Output distributionOutput unitResulting loss
Gaussian with fixed variance, $\mathcal{N}(y; f(x;\theta), I)$LinearMean squared error, up to scale and a constant [5]
Bernoulli over a binary labelSigmoidBernoulli negative log-likelihood (binary cross-entropy, log loss) [5]
Categorical over $n$ classesSoftmaxCategorical cross-entropy [5]
Gaussian mixtureSoftmax weights, means, variancesMixture negative log-likelihood [5]

Expanded article table

The softmax turns a vector of logits $z$ into a categorical distribution, $p(k \mid x) = \exp(z_k) / \sum_i \exp(z_i)$ [10]. Networks with Gaussian mixture outputs, often called mixture density networks (Jacobs et al., 1991; Bishop, 1994), are used when the target can have several peaks for the same input; their mixing weights usually come from a softmax so that they form a valid categorical distribution [5]. Practitioners also reshape target and output distributions on purpose:

  • Label smoothing. Szegedy and colleagues replaced the one-hot target with $q'(k) = (1 - \epsilon)\delta_{k,y} + \epsilon/K$, a mixture of the true label and a uniform distribution, to stop the model from becoming overconfident. With $K = 1000$ ImageNet classes and $\epsilon = 0.1$ they reported an improvement of about 0.2 percentage points in both top-1 and top-5 error [10]. See label smoothing.
  • Temperature. Dividing logits by a temperature $T$ before the softmax produces a softer distribution when $T > 1$. Knowledge distillation as described by Hinton, Vinyals and Dean trains a small model on these soft targets at a high temperature and uses $T = 1$ after training [11].

Language models

A statistical language model learns a probability distribution over sequences of words; Bengio and colleagues described this as learning "the joint probability function of sequences of words in a language" in their 2003 neural probabilistic language model [12]. Modern language models apply the chain rule and compute the probability of a text as the product of next-token conditionals, $P(x_{1:T}) = \prod_i P(x_i \mid x_1, \ldots, x_{i-1})$ [13]. Each step of generation is therefore a draw from, or a search over, a categorical distribution over the vocabulary.

Holtzman and colleagues found that decoding for maximum probability, as in beam search, produces bland and repetitive text even from strong models, and proposed nucleus (top-p) sampling, which samples from the smallest set of tokens whose cumulative probability reaches $p$, after renormalizing [13]. Top-k sampling instead keeps a fixed number $k$ of the most probable tokens. Sampling temperature rescales the logits before the softmax; values below 1 skew the distribution toward high-probability tokens and reduce the mass in the tail [13].

Generative models

Many deep generative models are defined by how they represent a distribution over data:

  • Variational autoencoders. Kingma and Welling's example model uses a standard isotropic Gaussian prior $\mathcal{N}(z; 0, I)$ over latent variables, a Gaussian (for real-valued data) or Bernoulli (for binary data) decoder, and a diagonal Gaussian approximate posterior, trained by maximizing a reparameterized variational lower bound [14]. See reparameterization trick.
  • Generative adversarial networks. A generator learns to capture the data distribution while a discriminator estimates whether a sample came from the data or the generator. Goodfellow and colleagues showed that, with an optimal discriminator, the generator's objective equals $-\log 4 + 2,\mathrm{JSD}(p_{\text{data}} ,|, p_g)$, minimized only when the generator's distribution equals the data distribution [15]. The Wasserstein GAN uses the Earth-Mover distance instead; its authors reported more stable training and fewer problems such as mode collapse [9].
  • Normalizing flows. Rezende and Mohamed described a normalizing flow as transforming a simple initial density into a more complex one through a sequence of invertible transformations [16]. Because the transformation is invertible, the change-of-variables formula gives the model density exactly, $p_X(x) = p_Z(f(x)) \lvert \det(\partial f(x) / \partial x^\top) \rvert$, which lets models such as Real NVP compute exact log-likelihoods and draw exact samples [17].
  • Diffusion models. Sohl-Dickstein and colleagues proposed slowly destroying structure in a data distribution with an iterative forward diffusion process and learning a reverse process that restores it [18]. In denoising diffusion probabilistic models, the forward process is a Markov chain that adds Gaussian noise according to a variance schedule, and each reverse step is a Gaussian with learned parameters [19]. Song and colleagues recast this as a stochastic differential equation that carries the data distribution to a known prior, with a reverse-time equation that depends only on the score, the gradient of the log density of the noised data distribution [20]. See score matching.

Reinforcement learning

In reinforcement learning, Bellemare, Dabney and Munos argued for modeling the full distribution of returns rather than only its expectation, the value. Their categorical algorithm represents the return distribution as a discrete distribution over a fixed set of $N$ atoms between a minimum and maximum value and trains it with the cross-entropy term of a KL divergence; the 51-atom agent was evaluated on games from the Arcade Learning Environment [21].

Software

Deep learning frameworks ship distribution libraries. PyTorch's torch.distributions package, which "generally follows the design of the TensorFlow Distributions package", provides parameterizable distributions and sampling functions, including Bernoulli, Beta, Binomial, Categorical, Dirichlet, Gamma, Laplace, Multinomial, MultivariateNormal, Normal, Poisson, Uniform and MixtureSameFamily [22]. It supports the two standard ways of differentiating through random samples: the score-function estimator (REINFORCE), which needs only sample() and log_prob() and underlies policy gradient methods, and the pathwise derivative through rsample(), the reparameterization trick common in variational autoencoders [22].

Technical subtleties

A fully rigorous treatment of continuous distributions uses measure theory. Without it, paradoxes can arise: it is possible to construct two disjoint sets whose probabilities, computed naively, add up to more than 1. Measure theory characterizes which sets can be assigned probabilities consistently and gives precise meaning to statements that hold "almost everywhere", that is, except on a set of measure zero [1].

Transforming a continuous variable also changes its density in a way that is easy to get wrong. If $y = g(x)$ for an invertible, differentiable $g$, then $p_y(y) \ne p_x(g^{-1}(y))$ in general; the correct formula multiplies by $\lvert \partial x / \partial y \rvert$, and in several dimensions by the absolute determinant of the Jacobian, to account for how $g$ stretches or compresses space [1]. This correction is the same one that normalizing flows exploit [17].

Continue reading

More from See also

More to read

Topics:Machine LearningMathematicsStatistics

References

  1. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10 ^11 ^12 ^13 ^14 ^15 ^16 ^17 ^18 ^19 ^20 ^21 ^22 ^23 ^24 ^25 ^26 ^27 ^28 ^29 ^30 ^31 ^32 ^33 ^34 ^35 ^36 ^37 ^38 ^39 ^40 ^41 ^42 ^43 ^44 ^45Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron. "Probability and Information Theory" (Chapter 3). Deep Learning, MIT Press, 2016. deeplearningbook.org/...prob
  2. ^1 ^2 ^3Hájek, Alan. "Interpretations of Probability". Stanford Encyclopedia of Philosophy, first published 2002-10-21, substantive revision 2023-11-16. plato.stanford.edu/...probability-interpret
  3. ^1 ^2 ^3NIST/SEMATECH. "1.3.6.2. Related Distributions". e-Handbook of Statistical Methods, National Institute of Standards and Technology. itl.nist.gov/...eda362
  4. ^1 ^2 ^3 ^4Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron. "Machine Learning Basics" (Chapter 5). Deep Learning, MIT Press, 2016. deeplearningbook.org/...ml
  5. ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9 ^10Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron. "Deep Feedforward Networks" (Chapter 6). Deep Learning, MIT Press, 2016. deeplearningbook.org/...mlp
  6. ^1 ^2NIST/SEMATECH. "1.3.6.6.18. Binomial Distribution". e-Handbook of Statistical Methods, National Institute of Standards and Technology. itl.nist.gov/...eda366i
  7. ^1 ^2NIST/SEMATECH. "1.3.6.6.19. Poisson Distribution". e-Handbook of Statistical Methods, National Institute of Standards and Technology. itl.nist.gov/...eda366j
  8. ^NIST/SEMATECH. "1.3.6.6.17. Beta Distribution". e-Handbook of Statistical Methods, National Institute of Standards and Technology. itl.nist.gov/...eda366h
  9. ^1 ^2 ^3Arjovsky, Martin; Chintala, Soumith; Bottou, Léon. "Wasserstein GAN". arXiv:1701.07875, 2017. arxiv.org/...1701.07875
  10. ^1 ^2Szegedy, Christian; Vanhoucke, Vincent; Ioffe, Sergey; Shlens, Jonathon; Wojna, Zbigniew. "Rethinking the Inception Architecture for Computer Vision". arXiv:1512.00567, 2015. arxiv.org/...1512.00567
  11. ^Hinton, Geoffrey; Vinyals, Oriol; Dean, Jeff. "Distilling the Knowledge in a Neural Network". arXiv:1503.02531, 2015. arxiv.org/...1503.02531
  12. ^Bengio, Yoshua; Ducharme, Réjean; Vincent, Pascal; Jauvin, Christian. "A Neural Probabilistic Language Model". Journal of Machine Learning Research 3:1137-1155, 2003. jmlr.org/...bengio03a
  13. ^1 ^2 ^3Holtzman, Ari; Buys, Jan; Du, Li; Forbes, Maxwell; Choi, Yejin. "The Curious Case of Neural Text Degeneration". ICLR 2020; arXiv:1904.09751. arxiv.org/...1904.09751
  14. ^Kingma, Diederik P.; Welling, Max. "Auto-Encoding Variational Bayes". arXiv:1312.6114, 2013. arxiv.org/...1312.6114
  15. ^1 ^2Goodfellow, Ian J.; Pouget-Abadie, Jean; Mirza, Mehdi; Xu, Bing; Warde-Farley, David; Ozair, Sherjil; Courville, Aaron; Bengio, Yoshua. "Generative Adversarial Networks". arXiv:1406.2661, 2014. arxiv.org/...1406.2661
  16. ^Rezende, Danilo Jimenez; Mohamed, Shakir. "Variational Inference with Normalizing Flows". arXiv:1505.05770, 2015. arxiv.org/...1505.05770
  17. ^1 ^2Dinh, Laurent; Sohl-Dickstein, Jascha; Bengio, Samy. "Density estimation using Real NVP". ICLR 2017; arXiv:1605.08803. arxiv.org/...1605.08803
  18. ^1 ^2Sohl-Dickstein, Jascha; Weiss, Eric A.; Maheswaranathan, Niru; Ganguli, Surya. "Deep Unsupervised Learning using Nonequilibrium Thermodynamics". arXiv:1503.03585, 2015. arxiv.org/...1503.03585
  19. ^Ho, Jonathan; Jain, Ajay; Abbeel, Pieter. "Denoising Diffusion Probabilistic Models". arXiv:2006.11239, 2020. arxiv.org/...2006.11239
  20. ^Song, Yang; Sohl-Dickstein, Jascha; Kingma, Diederik P.; Kumar, Abhishek; Ermon, Stefano; Poole, Ben. "Score-Based Generative Modeling through Stochastic Differential Equations". arXiv:2011.13456, 2020. arxiv.org/...2011.13456
  21. ^Bellemare, Marc G.; Dabney, Will; Munos, Rémi. "A Distributional Perspective on Reinforcement Learning". Proceedings of the 34th International Conference on Machine Learning, 2017; arXiv:1707.06887. arxiv.org/...1707.06887
  22. ^1 ^2PyTorch. "Probability distributions - torch.distributions". PyTorch 2.15 documentation. docs.pytorch.org/...distributions

Improve this article

Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.

2 revisions · v2 · 4,043 words · full history

Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify

Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here

Reviewer note: Independent fact-check before publication on November 18, 2026: 78 claims checked against 22 cited and independent sources; 0 material and 2 minor corrections applied before publication.

Cite this page: AI Wiki. "Probability distribution." aiwiki.ai, updated 18 Nov 2026, fact-checked 18 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/probability_distribution

Edit article