Stable Diffusion

Stable Diffusion is a family of generative image models that can synthesize and edit images from text and other conditions. The first public release, on August 22, 2022, combined research from the CompVis group at LMU Munich with support from Stability AI, Runway, and LAION.^[1][2][3] Its defining engineering choice was to run the generative process in a compressed image representation rather than directly over full-resolution pixels. This approach, called latent diffusion, reduced the computational cost of training and sampling while retaining a learned route back to pixel space.^[2]

The name now covers several related but non-interchangeable model lines. Stable Diffusion v1 and v2 use a convolutional U-Net denoiser. SDXL substantially enlarges and revises that design. Stable Diffusion 3 and 3.5 instead use multimodal diffusion transformers and a rectified-flow formulation. Checkpoints, text encoders, sampling settings, and licenses differ among releases, so results or terms for one release should not be assumed to apply to the entire family.

Development and scope

The original Stable Diffusion release grew out of the latent diffusion model described by Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer. That research showed how a pretrained image autoencoder could move diffusion training into a lower-dimensional latent space and how cross-attention could condition generation on text, semantic maps, or other inputs.^[2] The public v1 repository credited the CompVis researchers, Stability AI, Runway, and LAION. It described v1 as a factor-8 autoencoder paired with an 860-million-parameter U-Net and a frozen CLIP ViT-L/14 text encoder.^[3]

This page treats Stable Diffusion as an image-model family. It does not use the name as a synonym for every latent diffusion model, every product from Stability AI, or every community fine-tune. A fine-tuned checkpoint can inherit an official model's architecture without becoming an official Stable Diffusion release. Stable Video Diffusion is a related image-to-video family and is outside the main scope of this article.

How generation works

Diffusion in a latent space

A conventional diffusion model learns to reverse a process that gradually corrupts training data with noise. Denoising diffusion probabilistic models formalized one influential version of this procedure: training examples are perturbed at sampled noise levels, and a neural network learns the reverse denoising process.^[4]

Stable Diffusion v1 moves this operation into the latent space of a variational autoencoder. Its main stages are:

1. An encoder maps a training image from pixel space into a smaller latent tensor.
2. Noise is added to that tensor according to a training schedule.
3. A conditional denoising network learns to predict the added noise, or a related training target, from the noisy latent, the noise level, and the text condition.
4. At inference time, sampling starts from pseudorandom latent noise and repeatedly applies the denoiser according to a chosen scheduler.
5. The autoencoder decoder maps the final latent back to an image.

For v1, the autoencoder reduces both spatial dimensions by a factor of eight and produces four latent channels. The text prompt is encoded by a frozen CLIP ViT-L/14 model, and the non-pooled text representation enters the U-Net through cross-attention.^[5] Later Stable Diffusion lines retain the broad pattern of text conditioning and latent image generation, but they change the denoising architecture, text encoders, latent representation, or training formulation.

Latent diffusion is a computational compromise, not lossless compression. The decoder can discard or alter fine details, and generation can fail even if the text encoder represents the prompt well. The v1.4 model card specifically notes weaknesses in legible text, complex composition, faces and people, and non-English prompting, along with losses introduced by the autoencoder.^[5]

Text conditioning and guidance

The prompt does not specify pixels directly. A text encoder converts tokenized text into vectors, and cross-attention lets image features interact with those vectors during denoising. This makes prompt wording, tokenization, text-encoder choice, and context length part of the model's behavior. It also explains why a prompt tuned for v1 may behave differently with v2, SDXL, or SD3.

Stable Diffusion v1 was trained with some text conditioning deliberately dropped. That supports classifier-free guidance, which combines a conditional prediction with an unconditional prediction at sampling time.^[5][6] Increasing the guidance scale generally pushes a sample toward the condition more strongly, but it also changes the quality-diversity tradeoff and can produce artifacts. Guidance scale is therefore a sampling control, not a confidence score.

Text-to-image, image-to-image, and inpainting

In text-to-image generation, the initial latent is noise. In image-to-image generation, an input image is first encoded and then noised to a selected strength before denoising under a prompt. Inpainting adds a mask so that selected regions are regenerated while the remaining image supplies context.^[3] Stable Diffusion 2 also shipped separate depth-conditioned, inpainting, and upscaling models; these are task-specific checkpoints rather than abilities guaranteed by every base checkpoint.^[7]

A numeric seed initializes pseudorandom sampling, but a seed alone does not fully identify an output. Reproduction also depends on the exact checkpoint and component files, scheduler and step count, guidance settings, image dimensions, software implementation, numeric precision, and any adapters or refiners.

Model lines and releases

Stable Diffusion v1

The v1 series used a factor-8 autoencoder, a U-Net denoiser, and CLIP ViT-L/14 text conditioning. The public repository described training first at 256 by 256 pixels and then at 512 by 512 pixels.^[3] The v1.4 model card records a sequence of checkpoints trained on English-language subsets of LAION data, followed by aesthetic filtering and 512-pixel fine-tuning. It also records 10 percent conditioning dropout for v1.3 and v1.4 to support classifier-free guidance.^[5]

The August 2022 public release recommended v1.4 and distributed code and weights under the CreativeML Open RAIL-M license.^[1] Later v1 checkpoints must still be identified explicitly in reproducible work because a family name does not select one set of weights.

Stable Diffusion v2

Stable Diffusion 2.0 was released in November 2022. It replaced v1's CLIP text encoder with an OpenCLIP encoder and offered text-to-image checkpoints intended for 512 by 512 and 768 by 768 output, alongside separate depth, inpainting, and upscaling models.^[7] The release record says its text-to-image training data came from an aesthetic subset of LAION-5B with additional adult-content filtering.

Stable Diffusion 2.1 was a further fine-tune of the 2.0 768-pixel checkpoint. Its model card identifies OpenCLIP-ViT/H as the fixed text encoder and the CreativeML Open RAIL++-M license.^[8] Because the text encoder and data filtering differ from v1, prompt behavior and learned concepts can differ even when the sampling interface looks similar.

SDXL

SDXL retained latent diffusion and a U-Net but was a major architectural revision. Its technical report describes a U-Net backbone three times larger than those in earlier Stable Diffusion releases, a second text encoder, a larger cross-attention context, new conditioning variables, and training over multiple aspect ratios. It also introduced an optional refiner that performs later denoising as an image-to-image stage.^[9] The released base model uses OpenCLIP-ViT/G and CLIP ViT-L text encoders and can run without the refiner.^[9]

SDXL 1.0 was released on July 26, 2023, at a native 1024-by-1024-pixel scale.^[10] "SDXL" can refer to the base checkpoint, the optional base-plus-refiner pipeline, or the wider model line, so evaluations should state which configuration was used.

SDXL Turbo is a distilled SDXL variant rather than a replacement checkpoint for the base model. Its Adversarial Diffusion Distillation method combines a teacher signal based on score distillation with an adversarial loss to support one to four sampling steps.^[11] The model card identifies it as distilled from SDXL 1.0 and warns that the preferred 512 by 512 setting, short prompt encoding, and low-step design impose their own limitations.^[12]

Stable Diffusion 3 and 3.5

Stable Diffusion 3 changed the denoiser from a U-Net to a Multimodal Diffusion Transformer, or MMDiT, and used rectified flow. Its research paper describes separate parameter streams for image and text tokens with bidirectional information exchange between the modalities. The paper also changes how noise levels are sampled during training and reports scaling studies across model sizes.^[13]

Stable Diffusion 3 Medium, released as downloadable weights in June 2024, uses three fixed text encoders: OpenCLIP-ViT/G, CLIP-ViT/L, and T5-XXL. Its model card describes synthetic and filtered publicly available training data and several packaging choices that include or omit some text-encoder weights.^[14] Omitting T5-XXL reduces resource requirements but changes the pipeline and its output behavior.

Stable Diffusion 3.5 followed in October 2024 with Large and Large Turbo checkpoints; Medium followed later that month. Stability AI's release record distinguishes an 8.1-billion-parameter Large model, a four-step distilled Large Turbo, and a 2.5-billion-parameter Medium model using an MMDiT-X variant.^[15] The Large model card records query-key normalization and three text encoders.^[16] These details should not be retroactively attributed to SD3 Medium or earlier U-Net releases.

Scroll sideways for more →

The resolutions in this table describe the principal training or recommended generation scale in the cited release records. They are not hard output limits, and arbitrary dimensions can reduce quality or composition reliability.

Training data

The original model family is closely associated with LAION. The LAION-5B paper describes a collection of 5.85 billion CLIP-filtered image-text pairs, including 2.32 billion pairs classified as English.^[17] Stable Diffusion v1 did not train uniformly on all of LAION-5B. Its model card lists LAION-2B-en and progressively filtered high-resolution and aesthetic subsets for different checkpoints.^[5] Stable Diffusion 2 used a different aesthetic subset and additional filtering.^[7]

These records matter for three reasons. First, a model's behavior reflects both the broad source collection and the checkpoint-specific filters. Second, a caption is web-derived metadata, not a verified literal description of an image. Third, access to model weights does not provide a complete copy of the training images or resolve rights in the underlying data.

Later cards disclose training data at a higher level. The SD3 Medium card says training combined synthetic data with filtered publicly available data and describes separate pretraining, aesthetic fine-tuning, and preference data.^[14] The SD3.5 Large card similarly says it used synthetic and filtered publicly available data without publishing a row-level training manifest.^[16] These statements support only the disclosed categories and counts; they do not establish that every training item is known, licensed under the model license, or free of harmful material.

Licensing and distribution

Stable Diffusion does not have one family-wide license. The exact license file attached to a checkpoint controls the terms for that checkpoint and must be checked before use or redistribution.

The v1 weights use CreativeML Open RAIL-M. That license grants broad rights to use, reproduce, modify, and distribute the model and derivatives, but it also imposes use-based restrictions, redistribution conditions, attribution duties, and a requirement that model derivatives retain at least the stated use restrictions. It explicitly excludes the training data from the licensed material.^[18]

Stable Diffusion 2.1 and SDXL 1.0 use CreativeML Open RAIL++-M according to their model cards. The SDXL license likewise contains use restrictions and conditions for redistribution.^[19] These restrictions mean that "downloadable weights" is a more precise description than assuming the checkpoints carry an unrestricted software license.

The current repositories for SD3, SD3.5, and SDXL Turbo use the Stability AI Community License. As recorded at the July 2026 research cutoff, the official license page allows research and non-commercial use and permits commercial use without a fee for individuals or organizations below USD 1 million in annual revenue, subject to the agreement and acceptable-use terms. A separate enterprise license is required once annual revenue exceeds that threshold for commercial use.^[20] License terms and covered-model lists can change. This article summarizes the records but is not legal advice.

Model licenses also do not settle copyright, privacy, trademark, publicity, or other rights in prompts, input images, training data, or outputs. Those questions depend on the material, jurisdiction, and use.

Safety and limitations

Pipeline safeguards

The original CompVis reference script included a CLIP-based safety checker intended to reduce explicit outputs and an invisible watermarking component intended to mark generated images.^[3] The safety checker is a separate image-classification component in the pipeline, not a constraint encoded into every Stable Diffusion checkpoint. Its model card describes it as an NSFW image identifier and warns about biases and harmful stereotypes inherited from CLIP.^[21]

A packaged checker can miss unsafe content or flag benign content, and local pipelines can omit or replace it. Deployment safety therefore cannot be inferred from the checkpoint name alone. A public-facing system needs its own testing, input and output policies, abuse monitoring, and handling for false positives and false negatives.

Representation and prompt limitations

The v1.4 card says the model was trained primarily with English captions, performs worse on non-English prompts, and can reinforce social biases present in its data.^[5] A NeurIPS 2023 study evaluated DALL-E 2, Stable Diffusion v1.4, and Stable Diffusion v2 across prompts involving occupations and gender-coded adjectives. It reported systematic differences in represented gender and ethnicity within that study's generated samples.^[22] Those findings document the evaluated models and protocol; they should not be converted into a single bias score for every later checkpoint or prompt.

Common image failures include malformed anatomy, incorrect object counts, inconsistent spatial relations, illegible or misspelled text, and loss of fine identity details. The severity varies by checkpoint, prompt, sampler, and resolution. A visually plausible image is not evidence that depicted facts, people, places, documents, or events are real.

Memorization and privacy

Generative models can reproduce unusually memorized training examples under some conditions. Carlini and colleagues used a generate-and-filter attack to extract training examples from diffusion models they studied, including Stable Diffusion, and found that duplication in training data increased extraction risk.^[23] The result does not mean that an arbitrary training image can always be recovered, but it shows that a diffusion checkpoint should not be treated as a privacy guarantee.

Fine-tuning on personal or confidential images can add further privacy and consent risks. Operators should document data provenance, permission, retention, and deletion procedures before training or sharing an adapter or checkpoint.

Adaptation and ecosystem

Stable Diffusion's released weights and modular pipeline encouraged several forms of adaptation:

• Textual Inversion keeps the base text-to-image model frozen and learns a new token embedding from a small set of example images. Its original paper used three to five images to represent a user-supplied object or style as a new word-like embedding.^[24]
• DreamBooth fine-tunes a pretrained text-to-image model so that a rare identifier refers to a particular subject, with a prior-preservation objective intended to retain the subject class.^[25]
• LoRA freezes base weights and trains low-rank update matrices. The method was introduced for language-model adaptation; when the same mechanism is used with an image model, the targeted layers and compatible base checkpoint still have to be specified.^[26]
• ControlNet adds trainable control branches around a locked pretrained diffusion model. The original study tested Stable Diffusion with conditions such as edges, depth maps, segmentation maps, and human poses.^[27]

These artifacts are not interchangeable. A text embedding, LoRA adapter, ControlNet, full fine-tune, VAE, and base checkpoint modify different components and may target only one architecture or model line. Loading an adapter into a superficially similar but incompatible checkpoint can fail technically or produce unintended results.

Inference libraries package the text encoder, denoiser, autoencoder, scheduler, safety component, and post-processing into pipelines. Hugging Face's Diffusers repository explicitly treats these as separate components and supports interchangeable schedulers.^[28] Graphical interfaces and hosted services add workflow features, but an interface version is not a model version. Reproducible reports should name the underlying checkpoint and component revisions rather than only the application used.

Evaluation and reproducibility

A useful Stable Diffusion evaluation states the task and freezes the whole pipeline. At minimum, it should report:

• the exact checkpoint or repository revision;
• the VAE, text encoder, refiner, and adapter files;
• prompt and negative prompt text;
• scheduler or sampler, number of steps, and guidance setting;
• seed, image dimensions, batch size, and numeric precision;
• software versions and relevant hardware;
• any safety filtering, image selection, or post-processing;
• whether results were selected from multiple attempts.

Automated text-image similarity and image-quality metrics capture only parts of performance. The HEIM evaluation framework therefore separates 12 aspects, including alignment, quality, aesthetics, originality, reasoning, bias, toxicity, multilinguality, robustness, and efficiency.^[29] Human preference studies are also sensitive to prompt selection, rater instructions, sample filtering, and the compared inference budgets. Claims that one line "beats" another should therefore be tied to a named checkpoint, protocol, and source rather than generalized to the family.

References

1. ^Stability AI. "Stable Diffusion Public Release." August 22, 2022. stability.ai/...stable-diffusion-public-release
2. ^Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. "High-Resolution Image Synthesis With Latent Diffusion Models." CVPR 2022, pp. 10684-10695. openaccess.thecvf.com/...on_Models_CVPR_2022_paper
3. ^CompVis. "Stable Diffusion: A latent text-to-image diffusion model." GitHub repository and v1 release documentation. Accessed July 28, 2026. github.com/...stable-diffusion
4. ^Ho, J., Jain, A. N., and Abbeel, P. "Denoising Diffusion Probabilistic Models." Advances in Neural Information Processing Systems 33, 2020. proceedings.neurips.cc/...67f1ab10179ca4b-Abstract
5. ^CompVis. "Stable Diffusion v1-4 Model Card." Hugging Face. Accessed July 28, 2026. huggingface.co/...stable-diffusion-v1-4
6. ^Ho, J., and Salimans, T. "Classifier-Free Diffusion Guidance." 2022. arxiv.org/...2207.12598
7. ^Stability AI. "Stable Diffusion 2.0 Release." November 24, 2022. stability.ai/...stable-diffusion-v2-release
8. ^Stability AI. "Stable Diffusion v2-1 Model Card." Hugging Face. Accessed July 28, 2026. huggingface.co/...README.md
9. ^Podell, D., English, Z., Lacey, K., et al. "SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis." 2023. arxiv.org/...2307.01952
10. ^Stability AI. "Announcing SDXL 1.0." July 26, 2023. stability.ai/...stable-diffusion-sdxl-1-announcement
11. ^Sauer, A., Lorenz, D., Blattmann, A., and Rombach, R. "Adversarial Diffusion Distillation." 2023. arxiv.org/...2311.17042
12. ^Stability AI. "SDXL-Turbo Model Card." Hugging Face. Accessed July 28, 2026. huggingface.co/...sdxl-turbo
13. ^Esser, P., Kulal, S., Blattmann, A., et al. "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis." 2024. arxiv.org/...2403.03206
14. ^Stability AI. "Stable Diffusion 3 Medium Model Card." Hugging Face. Accessed July 28, 2026. huggingface.co/...stable-diffusion-3-medium
15. ^Stability AI. "Introducing Stable Diffusion 3.5." October 22, 2024, updated October 29, 2024. stability.ai/...introducing-stable-diffusion-3-5
16. ^Stability AI. "Stable Diffusion 3.5 Large Model Card." Hugging Face. Accessed July 28, 2026. huggingface.co/...stable-diffusion-3.5-large
17. ^Schuhmann, C., Beaumont, R., Vencu, R., et al. "LAION-5B: An open large-scale dataset for training next generation image-text models." Advances in Neural Information Processing Systems 35, Datasets and Benchmarks Track, 2022. papers.nips.cc/...Abstract-Datasets_and_Benchmarks
18. ^Rombach, R., Esser, P., and contributors. "CreativeML Open RAIL-M." August 22, 2022. github.com/...LICENSE
19. ^Stability AI. "CreativeML Open RAIL++-M License for SDXL 1.0." July 26, 2023. huggingface.co/...LICENSE.md
20. ^Stability AI. "Stability AI License." Accessed July 28, 2026. stability.ai/license
21. ^CompVis and Hugging Face. "Model Card for stable-diffusion-safety-checker." Hugging Face. Accessed July 28, 2026. huggingface.co/...stable-diffusion-safety-checker
22. ^Luccioni, A. S., Akiki, C., Mitchell, M., and Jernite, Y. "Stable Bias: Evaluating Societal Representations in Diffusion Models." Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track, 2023. proceedings.neurips.cc/...-Datasets_and_Benchmarks
23. ^Carlini, N., Hayes, J., Nasr, M., et al. "Extracting Training Data from Diffusion Models." 32nd USENIX Security Symposium, 2023, pp. 5253-5270. usenix.org/...carlini
24. ^Gal, R., Alaluf, Y., Atzmon, Y., et al. "An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion." 2022. arxiv.org/...2208.01618
25. ^Ruiz, N., Li, Y., Jampani, V., et al. "DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation." CVPR 2023. arxiv.org/...2208.12242
26. ^Hu, E. J., Shen, Y., Wallis, P., et al. "LoRA: Low-Rank Adaptation of Large Language Models." ICLR 2022. arxiv.org/...2106.09685
27. ^Zhang, L., Rao, A., and Agrawala, M. "Adding Conditional Control to Text-to-Image Diffusion Models." ICCV 2023, pp. 3836-3847. openaccess.thecvf.com/...on_Models_ICCV_2023_paper
28. ^Hugging Face. "Diffusers: diffusion models for image, video, and audio generation in PyTorch." GitHub repository. Accessed July 28, 2026. github.com/...diffusers
29. ^Lee, T., Yasunaga, M., Meng, C., et al. "Holistic Evaluation of Text-to-Image Models." Advances in Neural Information Processing Systems 36, Datasets and Benchmarks Track, 2023. papers.nips.cc/...Abstract-Datasets_and_Benchmarks