Diffusion model

A diffusion model is a generative model that learns to transform samples from a simple reference distribution into samples resembling a data distribution by reversing a gradual corruption process. In the common continuous-data formulation, a fixed forward process adds noise over a sequence of times, while a learned reverse process removes that noise. The first diffusion probabilistic model for deep learning was published in 2015.^[1] Denoising diffusion probabilistic models (DDPMs), introduced in 2020, made the approach practical for high-quality image synthesis by connecting a variational objective to denoising score matching.^[2]

The term is used at two levels. In a narrow sense, it means a probabilistic model with explicit forward and reverse diffusion processes. In a broader research usage, it can include closely related score-based models whose sampling dynamics are expressed as stochastic differential equations. Discrete diffusion models adapt the corruption-and-reversal idea to tokens or other finite state spaces. Flow-matching models can use diffusion probability paths, but flow matching is a broader framework and is not synonymous with diffusion modeling.

Diffusion models are used in computer vision, audio generation, language modeling, molecular modeling, protein design, and robot control. Their results depend on the training data, objective, architecture, conditioning method, and sampler. Performance claims therefore need to be tied to a particular model, dataset, metric, and evaluation protocol rather than treated as properties of every diffusion model.

Definition and scope

A diffusion model specifies a path between a data distribution and a tractable prior distribution, often a multivariate Gaussian. During training, examples can be corrupted directly to randomly chosen noise levels, so the complete forward chain does not need to be simulated for every update. A network learns information needed to reverse the corruption. At inference, sampling begins from the prior and applies a sequence of learned denoising updates or integrates corresponding continuous-time dynamics.^[2]

This article covers diffusion models as a general machine learning family. DDPM, DDIM, latent diffusion, the diffusion transformer, classifier-free guidance, consistency models, flow matching, discrete diffusion, and individual systems have separate articles. Those methods are discussed here only where they clarify the family as a whole.

The word "diffusion" does not mean that every implementation copies a physical diffusion equation exactly. It identifies a mathematical construction in which a simple corruption process and its learned reversal define a generative procedure. The exact state space, noise process, time parameterization, prediction target, and numerical solver vary among implementations.

Historical development

Aapo Hyvarinen introduced score matching in 2005 as a way to estimate an unnormalized statistical model by matching gradients of log densities. The objective avoids evaluating the model's normalization constant.^[3] In 2011, Pascal Vincent showed that a denoising autoencoder criterion could be interpreted as score matching against a noise-smoothed data distribution. This result connected learning to denoise with estimating the direction in which probability density increases.^[4]

Jascha Sohl-Dickstein and coauthors applied a gradual forward corruption and learned reverse process to deep generative modeling in 2015.^[1] The method established the probabilistic diffusion construction but did not by itself settle the architecture or training parameterization used by later systems.

In 2019, Yang Song and Stefano Ermon trained a noise-conditional score network across several Gaussian noise scales and sampled with annealed Langevin dynamics.^[5] In 2020, Jonathan Ho, Ajay Jain, and Pieter Abbeel introduced DDPMs. Their practical formulation trained a network with a weighted variational objective closely related to denoising score matching and Langevin dynamics.^[2]

A continuous-time account published at ICLR 2021 placed score-based and diffusion probabilistic models in a common framework. It described a forward stochastic differential equation (SDE), a reverse-time SDE determined by the time-dependent score, and a probability-flow ordinary differential equation (ODE) with the same marginal distributions.^[6] This framework explains why algorithms derived from different starting points can share training targets and sampling equations without being identical in every implementation.

Subsequent work separated and improved individual design choices. Nichol and Dhariwal learned reverse-process variances and reported that, for their evaluated models, this allowed sampling with about an order of magnitude fewer forward passes with little change in sample quality.^[7] Dhariwal and Nichol combined architectural changes with classifier guidance and reported stronger class-conditional ImageNet results than the comparison generative models used in their 2021 study.^[8] Those results were benchmark-specific historical findings, not proof that diffusion models always outperform other generative methods.

Core formulation

Forward process

For continuous data, a standard discrete-time DDPM defines a Markov chain that incrementally adds Gaussian noise. If $x_0$ is a data sample, one common transition is

$$q(x_t \mid x_{t-1}) = \mathcal{N}\left(x_t;\sqrt{1-\beta_t}\,x_{t-1},\beta_t I\right),$$

where $t$ indexes the noise level and $\beta_t$ is a prescribed variance schedule. Define $\alpha_t = 1-\beta_t$ and $\bar{\alpha}_t = \prod_{s=1}^{t}\alpha_s$. The resulting marginal can be sampled directly:

$$x_t = \sqrt{\bar{\alpha}_t}\,x_0 + \sqrt{1-\bar{\alpha}_t}\,\epsilon, \qquad \epsilon \sim \mathcal{N}(0,I).$$

This closed form permits training on a randomly chosen timestep without running every preceding transition. For a suitably chosen schedule and terminal time, the distribution of $x_T$ approaches the selected prior. It is more accurate to describe this terminal state as approximately distributed according to that prior than as literally containing no information under every finite schedule.

Learned reverse process

Generation requires transitions in the opposite direction. A DDPM commonly parameterizes them as Gaussian conditionals,

$$p_\theta(x_{t-1}\mid x_t) = \mathcal{N}\left(x_{t-1};\mu_\theta(x_t,t), \Sigma_\theta(x_t,t)\right).$$

The network may predict the added noise, an estimate of the clean sample, or another algebraically related target. In the noise-prediction form popularized by DDPM, a simplified objective is

$$L_{\mathrm{simple}} = \mathbb{E}_{x_0,t,\epsilon} \left[ \lVert \epsilon-\epsilon_\theta(x_t,t)\rVert_2^2 \right].$$

The original DDPM paper derived this parameterization from a variational bound and connected the prediction network to the score of the noisy data distribution.^[2] Different weighting, variance, and parameterization choices change optimization and numerical behavior, even when they describe related reverse processes.

Score and continuous-time views

The score of a density $p_t(x)$ is

$$s_t(x)=\nabla_x\log p_t(x).$$

It points locally toward increasing log density. A score network estimates this field at each noise level. In the SDE formulation, a forward process

$$d x = f(x,t)\,dt + g(t)\,d w$$

has a reverse-time SDE whose drift includes the score $s_t(x)$. Once the score is estimated, samples can be generated with numerical SDE solvers. The associated probability-flow ODE provides a deterministic trajectory with matching time marginals under the assumptions of the framework.^[6] A deterministic sampler is therefore possible, but this does not make the underlying training distribution or every diffusion implementation deterministic.

Sampling

The ancestral DDPM sampler applies learned reverse transitions one after another. Its sequential network evaluations can make inference expensive. The number of evaluations is not a fixed property of the model family: it depends on the training parameterization, schedule, solver, error tolerance, and desired quality.

DDIM constructs non-Markovian processes with the same DDPM training objective and permits deterministic sampling paths. Its authors reported a 10 to 50 times wall-clock speedup over their DDPM baseline in the experiments in the paper.^[9] The figure should not be generalized to arbitrary hardware, implementations, or later samplers.

Sampling can also be treated as numerical integration of the diffusion ODE. DPM-Solver analytically handles part of that ODE and approximates the remaining integral with dedicated high-order methods. Its authors reported useful samples in 10 to 20 function evaluations on the datasets and models they tested.^[10] Solver comparisons remain sensitive to implementation, evaluation budget, and guidance strength.

Conditioning and guidance

Classifier guidance modifies a diffusion score with gradients from a separately trained classifier. The 2021 guided-diffusion study used this technique to trade sample diversity for fidelity on class-conditional image generation.^[8] The method depends on an additional classifier that can operate on noisy inputs.

Classifier-free guidance instead trains conditional and unconditional behavior together, commonly by dropping the condition for some training examples. At sampling time, the conditional and unconditional predictions are combined. Its authors described the method as obtaining a sample-quality and diversity tradeoff without a separate classifier.^[11] Increasing guidance is not a free improvement: it changes the sampled distribution and can reduce diversity or amplify artifacts.

A diffusion model can be unconditional or conditioned on information such as a class label, text representation, partial observation, low-resolution input, or geometric constraint. Conditioning information can enter the denoising network directly. In image models, cross-attention is one way to connect a spatial denoiser to text or other sequences.^[12]

Conditional generation does not guarantee that a requested constraint will be satisfied. The result depends on the conditioning representation, the training distribution, the guidance method, and the sampling settings. Applications that require exact physical, chemical, or safety constraints normally need task-specific validation outside the generative model.

Architectures and representation spaces

Diffusion is a probabilistic framework, not a single network architecture. Image DDPMs often use an encoder-decoder denoiser with multiscale features and skip connections. Other state spaces and modalities require different equivariances, token representations, or temporal structures.

Pixel-space models diffuse the observed array directly. This keeps the corruption process close to the data representation but makes the denoiser operate at full spatial resolution. Latent diffusion first uses an autoencoder to map an image into a lower-dimensional representation and runs diffusion there. The 2022 latent-diffusion paper reported a useful compromise between computational reduction and reconstruction detail and used cross-attention for conditioning.^[12] The decoder limits what can be reconstructed, so latent-space efficiency comes with an information-bottleneck tradeoff. Stable Diffusion is a prominent system built from the latent-diffusion approach, but it is one implementation rather than a synonym for diffusion models.

The denoising backbone can also be a transformer. The 2023 Diffusion Transformer study replaced the commonly used U-Net backbone with a transformer operating on latent image patches. Within the model family and compute range tested, increased forward-pass compute correlated with lower FID.^[13] This finding supports the viability of transformer denoisers but does not establish a universal scaling law across datasets, objectives, or modalities.

The model representation should match the geometry of the task. Images can use Euclidean pixel or latent tensors, molecules may require translations, rotations, and torsions, and protein backbones can require rotation-equivariant coordinates. Calling all of these "noise prediction" conceals important differences in state space and corruption kernel.

Noise schedules determine how signal-to-noise ratio changes with time. Reverse-variance choices determine how much stochasticity remains in each update. The EDM study separated noise levels, network preconditioning, loss weighting, and sampler design, then evaluated combinations of those choices.^[14] Its main general contribution was a modular design analysis. The paper's reported FID values belong to its specific benchmarks and configurations.

Faster generation and adjacent model families

Acceleration methods fall into several categories: changing the reverse variances, using a shorter or non-Markovian sampling path, applying a higher-order solver, distilling a multistep model, or training a model designed for few-step generation. These methods are not interchangeable, and quality at a small evaluation budget must be measured on the actual model and task.

Consistency models learn to map points on a probability-flow trajectory to its origin. They can be distilled from a diffusion model or trained independently, and they support one-step or multistep sampling.^[15] A consistency model trained on its own is a distinct generative family, even though its formulation and evaluation are closely connected to diffusion trajectories.

Flow matching trains a continuous normalizing flow by regressing a vector field associated with selected conditional probability paths. Gaussian diffusion paths are included as special cases, but flow matching also supports paths that are not diffusion paths, such as the optimal-transport interpolants studied in the original paper.^[16] It is therefore inaccurate to label every flow-matching model a diffusion model or to claim that the two frameworks are universally equivalent.

Discrete diffusion

Gaussian noising is natural for continuous vectors but not for categorical tokens. Discrete diffusion defines a forward corruption process with transition matrices over a finite state space. Structured Denoising Diffusion Probabilistic Models explored kernels based on uniform replacement, neighborhood structure, and absorbing states such as a mask token.^[17]

Later work developed objectives tailored to discrete scores and masked corruption. SEDD introduced score entropy to estimate ratios of probabilities in discrete spaces and evaluated the method on language modeling.^[18] Masked diffusion language models derived a simplified objective that is a mixture of masked-language-model losses and reported results approaching autoregressive perplexity on the benchmarks in that study.^[19]

These papers show that diffusion-like generation is not restricted to Gaussian image noise. They do not establish that diffusion language models always match or outperform an autoregressive model. Text quality, likelihood, sampling cost, sequence length, and infilling capability measure different properties and can favor different methods.

Applications

Diffusion models have been adapted by choosing a state representation, corruption process, network, and conditioning scheme appropriate to the domain.

Scroll sideways for more →

Success in one domain does not transfer automatically to another. A protein-design model, an image generator, and a robotic policy may share iterative denoising mathematics while differing in data, geometry, architecture, evaluation, and acceptable failure modes.

Evaluation

Evaluation should distinguish at least four questions:

• Distributional fit: likelihood or variational-bound estimates assess probabilistic fit under a specified formulation.
• Sample fidelity and coverage: image metrics such as FID, precision, and recall compare generated and reference feature distributions, but results depend on the feature extractor, sample count, preprocessing, and dataset.
• Conditional correctness: text-image alignment, constraint satisfaction, docking geometry, or task success requires domain-specific tests.
• Efficiency: training compute, memory, latency, number of network evaluations, and energy use are separate measurements.

No single metric establishes that one generative family is best in general. Comparisons should keep data, resolution, conditioning, architecture scale, sampling budget, and evaluation code as consistent as possible. Historical claims of "state of the art" are snapshots tied to a benchmark and publication date.

For stochastic generators, repeated sampling matters. Reporting only selected examples cannot estimate coverage or failure rates. For scientific and control applications, generated candidates require validation by appropriate simulations, experiments, or real-world task tests.

Limitations and risks

Iterative denoising usually requires multiple sequential network evaluations. Faster solvers or distilled models reduce that cost, but the quality-speed tradeoff is model- and task-dependent.^{[7][9][10][15]} Guidance can improve adherence or measured fidelity while reducing diversity.^[8][11] Latent models reduce denoising cost but inherit reconstruction limits from the encoder and decoder.^[12]

Training-data memorization is also an empirical privacy and provenance risk. Carlini and coauthors extracted more than 1,000 training examples from the image diffusion models and settings they studied.^[25] A separate CVPR study developed retrieval methods for detecting replicated training content and found replication in several evaluated datasets and models.^[26] These results show that diffusion training does not by itself prevent memorization. They do not imply that every output is copied, and they do not determine the copyright status of any particular model or output.

Diffusion models learn from their training distributions. Underrepresentation, label imbalance, caption errors, and other data defects can affect generated outputs. In controlled image experiments, class-imbalanced training data reduced both fidelity and diversity, especially for tail classes.^[27] That study establishes a failure mode in its tested settings, not a numerical prediction for every dataset or model.

The learned reverse process is approximate, and numerical sampling adds further approximation. Outputs can be implausible, internally inconsistent, or unsafe even when they score well on aggregate metrics. Deployment therefore requires evaluation for the specific domain, population, conditioning interface, and consequence of error.

References

1. ^Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. "Deep Unsupervised Learning using Nonequilibrium Thermodynamics." Proceedings of the 32nd International Conference on Machine Learning, 2015. proceedings.mlr.press/...sohl-dickstein15
2. ^Jonathan Ho, Ajay N. Jain, and Pieter Abbeel. "Denoising Diffusion Probabilistic Models." Advances in Neural Information Processing Systems 33, 2020. proceedings.neurips.cc/...67f1ab10179ca4b-Abstract
3. ^Aapo Hyvarinen. "Estimation of Non-Normalized Statistical Models by Score Matching." Journal of Machine Learning Research 6, 2005. jmlr.org/...hyvarinen05a
4. ^Pascal Vincent. "A Connection Between Score Matching and Denoising Autoencoders." Neural Computation 23(7), 2011. direct.mit.edu/...7677
5. ^Yang Song and Stefano Ermon. "Generative Modeling by Estimating Gradients of the Data Distribution." Advances in Neural Information Processing Systems 32, 2019. proceedings.neurips.cc/...1a96dcd947c7d93-Abstract
6. ^Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. "Score-Based Generative Modeling through Stochastic Differential Equations." International Conference on Learning Representations, 2021. openreview.net/forum
7. ^Alexander Quinn Nichol and Prafulla Dhariwal. "Improved Denoising Diffusion Probabilistic Models." Proceedings of the 38th International Conference on Machine Learning, 2021. proceedings.mlr.press/...nichol21a
8. ^Prafulla Dhariwal and Alexander Nichol. "Diffusion Models Beat GANs on Image Synthesis." Advances in Neural Information Processing Systems 34, 2021. proceedings.neurips.cc/...d77d02681df5cfa-Abstract
9. ^Jiaming Song, Chenlin Meng, and Stefano Ermon. "Denoising Diffusion Implicit Models." International Conference on Learning Representations, 2021. arxiv.org/...2010.02502
10. ^Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. "DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps." Advances in Neural Information Processing Systems 35, 2022. proceedings.neurips.cc/...c59e-Abstract-Conference
11. ^Jonathan Ho and Tim Salimans. "Classifier-Free Diffusion Guidance." arXiv, 2022. arxiv.org/...2207.12598
12. ^Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. "High-Resolution Image Synthesis With Latent Diffusion Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. openaccess.thecvf.com/...on_Models_CVPR_2022_paper
13. ^William Peebles and Saining Xie. "Scalable Diffusion Models with Transformers." Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023. openaccess.thecvf.com/...nsformers_ICCV_2023_paper
14. ^Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. "Elucidating the Design Space of Diffusion-Based Generative Models." Advances in Neural Information Processing Systems 35, 2022. proceedings.neurips.cc/...87eb694d946ce6b-Abstract
15. ^Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. "Consistency Models." Proceedings of the 40th International Conference on Machine Learning, 2023. proceedings.mlr.press/...song23a
16. ^Yaron Lipman, Ricky T. Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. "Flow Matching for Generative Modeling." International Conference on Learning Representations, 2023. openreview.net/forum
17. ^Jacob Austin, Daniel D. Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. "Structured Denoising Diffusion Models in Discrete State-Spaces." Advances in Neural Information Processing Systems 34, 2021. proceedings.neurips.cc/...e97125b70e6973d-Abstract
18. ^Aaron Lou, Chenlin Meng, and Stefano Ermon. "Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution." Proceedings of the 41st International Conference on Machine Learning, 2024. proceedings.mlr.press/...lou24a
19. ^Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin T. Chiu, Alexander Rush, and Volodymyr Kuleshov. "Simple and Effective Masked Diffusion Language Models." Advances in Neural Information Processing Systems 37, 2024. proceedings.neurips.cc/...e0ad-Abstract-Conference
20. ^Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D. Plumbley. "AudioLDM: Text-to-Audio Generation with Latent Diffusion Models." Proceedings of the 40th International Conference on Machine Learning, 2023. proceedings.mlr.press/...liu23f
21. ^Andreas Blattmann et al. "Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets." arXiv, 2023. arxiv.org/...2311.15127
22. ^Gabriele Corso, Hannes Stark, Bowen Jing, Regina Barzilay, and Tommi Jaakkola. "DiffDock: Diffusion Steps, Twists, and Turns for Molecular Docking." International Conference on Learning Representations, 2023. arxiv.org/...2210.01776
23. ^Joseph L. Watson et al. "De novo design of protein structure and function with RFdiffusion." Nature 620, 2023. nature.com/...s41586-023-06415-8
24. ^Cheng Chi, Siyuan Feng, Yilun Du, Zhenjia Xu, Eric Cousineau, Benjamin C. M. Burchfiel, and Shuran Song. "Diffusion Policy: Visuomotor Policy Learning via Action Diffusion." Robotics: Science and Systems XIX, 2023. roboticsproceedings.org/...p026
25. ^Nicholas Carlini et al. "Extracting Training Data from Diffusion Models." 32nd USENIX Security Symposium, 2023. usenix.org/...carlini
26. ^Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. "Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. openaccess.thecvf.com/...Diffusion_CVPR_2023_paper
27. ^Yiming Qin, Huangjie Zheng, Jiangchao Yao, Mingyuan Zhou, and Ya Zhang. "Class-Balancing Diffusion Models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023. openaccess.thecvf.com/...on_Models_CVPR_2023_paper