Variational Autoencoder

A variational autoencoder (VAE) is a latent-variable generative model that pairs a probabilistic decoder with a learned approximation to posterior inference. It is trained by maximizing a tractable lower bound on the data log-likelihood. The encoder-shaped inference network and the reparameterization estimator make this objective practical to optimize with neural networks and minibatches. Kingma and Welling introduced the auto-encoding variational Bayes method, while Rezende, Mohamed, and Wierstra developed a closely related stochastic backpropagation method at about the same time.^[1][2]

Despite its name and familiar encoder-decoder diagram, a VAE is not just an autoencoder with noise added. A deterministic autoencoder learns a point-valued code for reconstruction. A VAE specifies a joint probability model, an approximate posterior distribution for each observation, and a prior from which new latent values can be sampled. This probabilistic formulation connects Variational Inference, representation learning, and neural generative modeling.^[1][3]

Background and motivation

For a latent-variable model, the marginal likelihood of an observation $x$ is

$$p_\theta(x)=\int p_\theta(x,z)\,dz =\int p(z)p_\theta(x\mid z)\,dz.$$

The posterior $p_\theta(z\mid x)$ is generally difficult to evaluate because it contains the same marginal likelihood integral. Classical variational methods introduce a tractable distribution for each data point and optimize its parameters separately. A VAE instead learns an inference network $q_\phi(z\mid x)$ that predicts those variational parameters from $x$. This reuse of one inference model across observations is called amortized inference.^[2][3]

Amortization changes the computational problem. Once the network is trained, an approximate posterior can be obtained with a forward pass rather than a fresh optimization for every observation. The result is useful both as an approximate Bayesian inference procedure for the model and as an encoder that maps observations to distributions in a lower-dimensional space.^[2][3]

Formulation and architecture

Generative model

The standard VAE begins with a prior over latent variables, commonly

$$p(z)=\mathcal{N}(0,I),$$

and a decoder likelihood $p_\theta(x\mid z)$. A Neural Network with parameters $\theta$ produces the likelihood parameters. For example, it may output Bernoulli probabilities for binary data, a mean and variance for a Gaussian observation model, categorical probabilities for discrete sequences, or parameters of a count distribution for count data. The decoder is therefore a conditional probability model, not merely a function that returns a reconstruction.^[1][3]

Generation uses the directed model:

1. Draw $z\sim p(z)$.
2. Draw $x\sim p_\theta(x\mid z)$, or use a representative statistic such as the decoder mean.

This distinction matters. A generated sample from the specified likelihood can differ from the decoder's mean, and the chosen likelihood defines what the reconstruction term in training actually measures.^[3]

Inference model

The inference network $q_\phi(z\mid x)$, also called the recognition model or encoder, approximates the intractable posterior $p_\theta(z\mid x)$. A common continuous-latent choice is a diagonal Gaussian:

$$q_\phi(z\mid x) =\mathcal{N}\!\left(z;\mu_\phi(x), \operatorname{diag}\left(\sigma_\phi^2(x)\right)\right).$$

The network normally emits $\mu_\phi(x)$ and $\log \sigma_\phi^2(x)$. Using log variance permits an unconstrained network output, after which exponentiation produces a positive variance. The diagonal assumption is computationally convenient, but it cannot directly represent posterior correlations or multiple separated modes.^[1][3]

Encoder and decoder are roles in the probability model rather than fixed architectural types. Images commonly use convolutional or transformer blocks. Sequence VAEs have used a Recurrent Neural Network in one or both roles. Domain-specific likelihoods and conditioning variables often matter at least as much as the choice of backbone.^[7][22]

Evidence lower bound

For any approximate posterior $q_\phi(z\mid x)$, the data log-likelihood can be decomposed as

$$\log p_\theta(x) =\mathcal{L}(\theta,\phi;x) +D_{\mathrm{KL}}\!\left( q_\phi(z\mid x)\,\|\,p_\theta(z\mid x) \right),$$

where the evidence lower bound (ELBO) is

$$\mathcal{L}(\theta,\phi;x) =\mathbb{E}_{q_\phi(z\mid x)} \left[\log p_\theta(x\mid z)\right] -D_{\mathrm{KL}}\!\left( q_\phi(z\mid x)\,\|\,p(z) \right).$$

The Kullback-Leibler divergence in the first equation is nonnegative, so $\mathcal{L}\leq \log p_\theta(x)$. The gap is exactly the divergence from the approximate posterior to the model posterior. Maximizing the ELBO therefore trains the generative model while also fitting the inference network.^[3]

The two terms have distinct roles:

Scroll sideways for more →

It is common to call the first term a reconstruction term and the second a regularizer. That language is useful, but incomplete. Both are parts of one variational likelihood objective. The KL term does not guarantee that every latent point decodes to realistic data, nor does it guarantee a semantically organized representation. It only penalizes a particular divergence under the distributions and model being optimized.^[3]

Information and the aggregate posterior

Let the empirical data distribution be $p_{\mathrm{data}}(x)$, and define the aggregate posterior

$$q_\phi(z)= \mathbb{E}_{p_{\mathrm{data}}(x)} \left[q_\phi(z\mid x)\right].$$

The average per-observation KL can be decomposed into mutual information between data and latent variables plus a divergence between the aggregate posterior and the prior:

$$\mathbb{E}_{p_{\mathrm{data}}(x)} \left[ D_{\mathrm{KL}}(q_\phi(z\mid x)\|p(z)) \right] =I_q(x;z)+D_{\mathrm{KL}}(q_\phi(z)\|p(z)).$$

This identity clarifies two effects that the single KL term combines. It limits how much information the latent variable carries about individual observations, and it pressures the collection of encoded observations toward the prior. A model can reconstruct well by using substantial information while matching the prior poorly, or it can match the prior while carrying too little information for a downstream task. The decomposition also motivates objectives that weight mutual information, total correlation, or aggregate-posterior matching differently.^[10][13]

The information view is sometimes described as a rate-distortion tradeoff. The expected negative log-likelihood acts as a distortion defined by the observation model, while the KL contribution acts as a rate-like cost for the latent representation. Changing latent dimension, decoder capacity, likelihood variance, or a KL coefficient can move the operating point. There is no single best point independent of whether the goal is density estimation, faithful reconstruction, compression, controllable representation, or prediction.^[3][12][23]

Closed-form Gaussian KL

If $q_\phi(z\mid x)$ is a diagonal Gaussian and $p(z)=\mathcal{N}(0,I)$, the KL term for a $d$-dimensional latent variable is

$$D_{\mathrm{KL}}\!\left(q_\phi(z\mid x)\,\|\,p(z)\right) =-\frac{1}{2}\sum_{j=1}^{d} \left(1+\log\sigma_j^2-\mu_j^2-\sigma_j^2\right).$$

This term can be evaluated exactly for each observation. The expected log-likelihood is usually estimated with Monte Carlo samples from $q_\phi(z\mid x)$.^[1]

Reparameterization

Direct sampling appears to interrupt ordinary backpropagation because a sampled $z$ is a random node. For a diagonal Gaussian, the sample can instead be written as

$$\epsilon\sim\mathcal{N}(0,I),\qquad z=\mu_\phi(x)+\sigma_\phi(x)\odot\epsilon.$$

Randomness is now isolated in $\epsilon$, whose distribution does not depend on $\phi$. For a fixed draw of $\epsilon$, $z$ is a differentiable function of the encoder output. This pathwise gradient estimator lets gradients from the likelihood term pass through $z$ into the encoder.^[1][4]

Reparameterization is one member of a broader family of Monte Carlo gradient estimators. Pathwise estimators are often attractive for continuous variables, while score-function estimators apply more broadly but can have high variance. The correct estimator depends on the distribution, objective, and differentiability of the sampling transformation.^[4]

Training and use

A basic minibatch update is:

1. Compute $\mu_\phi(x)$ and $\log\sigma_\phi^2(x)$.
2. Sample $\epsilon$ and construct $z$ by reparameterization.
3. Evaluate $\log p_\theta(x\mid z)$.
4. Evaluate the KL term, analytically when possible.
5. Minimize the negative ELBO with Gradient Descent or an adaptive stochastic optimizer.

Training jointly updates $\theta$ and $\phi$. Reconstruction commonly uses a sample from $q_\phi(z\mid x)$; a deterministic visualization often decodes $\mu_\phi(x)$. Unconditional generation instead samples from $p(z)$. Interpolation is a diagnostic of the learned representation, not proof that intermediate points have a particular semantic meaning.^[1][3]

Posterior collapse

Posterior collapse occurs when the learned approximate posterior is close to the prior for many observations and the decoder makes little or no use of $z$. In the limiting case, $q_\phi(z\mid x)=p(z)$, the KL contribution is zero, and the latent variable carries no information about $x$. A sufficiently expressive autoregressive decoder can model local or sequential structure while ignoring the latent input, making this failure especially visible in text VAEs.^[5][7]

One explanation is an optimization imbalance: early in training, the inference network can lag behind the changing model posterior, and the decoder learns an easier solution that does not depend on $z$. He and colleagues found that more aggressive inference-network updates could reduce collapse in their experiments.^[5] Other interventions include:

Scroll sideways for more →

KL warm-up was used in Ladder VAE experiments, where the KL contribution was turned on gradually, and Bowman and colleagues combined annealing with word dropout for sentence generation.^[6][7] These methods should be treated as diagnostics and design choices rather than universal cures. A low KL can reflect collapse, an unnecessarily large latent space, or a model that genuinely needs little latent information.

Other failure modes

Approximation and amortization gaps

The ELBO gap is the divergence $D_{\mathrm{KL}}(q_\phi(z\mid x)\|p_\theta(z\mid x))$. Part of that gap can come from the selected variational family. A diagonal Gaussian cannot exactly represent a correlated, skewed, or multimodal posterior. This is often called an approximation gap. A second part can arise because the shared inference network does not find the best member of that family for every observation. That is an amortization gap.^[3]

A larger encoder can reduce amortization error without changing the posterior family. A flow can make the family more expressive. More per-observation optimization can test how much error comes from amortization, but sacrifices the fast forward-pass inference that defines the usual VAE workflow. Importance weighting changes the bound and training signal. These interventions answer different diagnoses and should not be treated as synonyms.^[3][8][9]

Prior mismatch and poor prior samples

Good reconstructions use latent values drawn from $q_\phi(z\mid x)$. Unconditional generation uses latent values drawn from $p(z)$. If the aggregate posterior covers the prior unevenly, the decoder can reconstruct training-like observations yet behave poorly for some prior draws. This is the more precise version of the informal "holes in latent space" problem.^[3][10]

The per-observation KL discourages mismatch but does not eliminate it in a finite, imperfectly optimized model. A flexible learned prior such as VampPrior can better follow the aggregate encoded distribution in some settings. Hierarchical priors, mixtures, and two-stage schemes are other possibilities. Each adds parameters or training complexity and can make ancestral sampling more expensive.^[10][17]

Likelihood mismatch and reconstruction artifacts

A decoder trained with a simple factorized Gaussian or Bernoulli likelihood may assign independent conditional distributions to pixels or features that are strongly dependent in the real data. When several fine-scale outcomes are plausible, optimizing a pixel-space likelihood can produce a conditional mean that looks smooth or blurry even if the probabilistic objective is behaving as specified. This is not caused solely by stochastic sampling or the word "variational"; it reflects the likelihood, architecture, bottleneck, and evaluation criterion together.^[3][19]

Perceptual and adversarial reconstruction objectives can preserve visually important structure better than a simple pixel loss, as in the first stage of latent diffusion. They also move the training objective away from a plain, fully specified observation likelihood. The result may be preferable for downstream visual generation while no longer supporting the same direct likelihood interpretation. The design should state this tradeoff rather than calling one scalar "reconstruction loss" without qualification.^[19]

Optimization and numerical problems

Gaussian VAEs commonly predict log variance because variance must remain positive and can span many scales. Very small predicted variance can produce large or unstable terms in the objective, while very large variance can inject destructive noise. Deep hierarchical models add many KL contributions, stochastic groups, and long computational paths. NVAE reported that residual posterior parameterization and spectral regularization were important to stable training in its deep image hierarchy.^[17]

The scales of the likelihood and KL also depend on reduction conventions. Summing a pixel likelihood over thousands of dimensions and averaging a KL over the batch is not equivalent to averaging both over every element. A reported $\beta$ value has no portable meaning unless preprocessing, likelihood, dimensional reductions, and batch reductions are also specified.

Representation claims without downstream evidence

Latent traversals can reveal local behavior, but their appearance is sensitive to coordinate choice. Interpolation can look smooth even when the representation is not aligned with real causal factors. Conversely, a representation useful for prediction need not have independently interpretable coordinates. The unsupervised disentanglement impossibility result makes the central point: semantic factor recovery requires assumptions about the model, data, or supervision.^[14]

For a representation claim, evaluation should match the claim. Factor recovery requires appropriate ground truth or interventions. Compression requires rate and distortion measurement. Scientific inference requires calibrated domain-specific checks. Recommendation requires held-out ranking. A visually pleasing two-dimensional plot cannot substitute for these tests.^[14][22][23]

Comparisons and tradeoffs

Deterministic autoencoder

Scroll sideways for more →

A VAE's prior gives it a defined generative procedure, but prior samples are good only if training makes the decoder and latent distribution work well together. The common shorthand that an ordinary autoencoder has "holes" while a VAE has a completely filled latent space is too strong. Aggregate-posterior mismatch, unused regions, and poor prior samples can remain in a trained VAE.^[3][10]

Other generative model families

The main distinction is not a permanent ranking of sample quality. It is the model and training objective:

Scroll sideways for more →

These boundaries can overlap. Diffusion probabilistic models can be written as deep hierarchical latent-variable models with a variational bound, and one analysis explicitly views them as a type of very deep VAE.^[27] Latent diffusion adds a separate autoencoding stage so the denoising model runs in a compressed representation.^[19] These mathematical and architectural connections do not make a one-step VAE decoder and an iterative diffusion sampler operationally identical.

Major extensions

VAE research has modified the conditioning variables, bound, posterior family, prior, hierarchy, and latent representation. Representative variants include:

Scroll sideways for more →

Conditional VAE

In a conditional VAE, the generative process can be written as

$$z\sim p_\theta(z\mid c),\qquad y\sim p_\theta(y\mid z,c),$$

with an inference model such as $q_\phi(z\mid y,c)$. The original CVAE work used this construction for structured prediction, where one input can have several plausible outputs. Conditioning can represent labels, observations, attributes, or other context, but it does not by itself guarantee controllability outside the training distribution.^[11]

Importance weighting, flows, and learned priors

The importance-weighted autoencoder replaces the single-sample ELBO with

$$\mathcal{L}_K(x)= \mathbb{E}_{z_{1:K}\sim q_\phi(z\mid x)} \left[ \log\frac{1}{K}\sum_{k=1}^{K} \frac{p_\theta(x,z_k)}{q_\phi(z_k\mid x)} \right].$$

Under the conditions analyzed in the original work, increasing $K$ tightens the bound toward the log-likelihood. More samples also increase computation, and a tighter bound does not automatically solve every inference or representation problem.^[8]

Normalizing flows increase posterior flexibility by passing a simple random variable through a sequence of invertible transformations and accounting for the Jacobian determinant. VampPrior changes a different component: it approximates a flexible prior with a mixture of encoder distributions conditioned on learned pseudo-inputs. Both address restrictions of the standard diagonal-Gaussian setup, but at different points in the model.^[9][10]

Disentanglement objectives

beta-VAE applies a coefficient to the KL term:

$$\mathcal{L}_{\beta} =\mathbb{E}_{q_\phi(z\mid x)}[\log p_\theta(x\mid z)] -\beta D_{\mathrm{KL}}(q_\phi(z\mid x)\|p(z)).$$

Increasing $\beta$ restricts latent information more strongly and can trade reconstruction fidelity for a more factorized representation. beta-TCVAE refined the analysis by decomposing the average KL contribution into index-code mutual information, total correlation, and dimension-wise KL, then emphasizing the total-correlation term.^[12][13]

Claims of automatic unsupervised disentanglement require an important caveat. Locatello and colleagues proved non-identifiability for general unsupervised settings without inductive biases on models and data, and their large empirical study found strong sensitivity to seeds and hyperparameters. A factorized prior or a larger KL weight alone does not identify the real generative factors in arbitrary data.^[14]

Discrete and hierarchical descendants

VQ-VAE replaces a continuous Gaussian bottleneck with nearest-neighbor lookup in a learned codebook. Its encoder receives a straight-through gradient, while codebook and commitment terms train the quantized representation. The original paper reported that this construction avoided posterior collapse in its tested settings and used a separate learned prior over discrete codes.^[15] VQ-VAE-2 later arranged discrete codes hierarchically and paired them with autoregressive priors for high-fidelity image generation.^[16]

Because its objective is not the ordinary Gaussian ELBO, VQ-VAE is best described as a closely related discrete autoencoding lineage. That lineage became important for tokenization. DALL-E, for example, trained a discrete VAE to map each $256\times256$ image to a $32\times32$ grid of image tokens, then modeled text and image tokens with an autoregressive transformer.^[18] This is strong evidence for a VAE-derived tokenizer role, not evidence that every modern tokenizer uses the same objective.

NVAE demonstrates a different direction. It retained the continuous VAE objective while scaling a deep hierarchy using multi-scale latent groups, depthwise separable convolutions, residual parameterization of approximate posteriors, and spectral regularization. Its results showed that architectural and optimization work could substantially strengthen likelihood-based image VAEs.^[17]

Applications and modern role

VAEs have served both as end-to-end generative models and as probabilistic representation components. The strongest application claims are specific systems and experiments, not an assertion that VAEs are universally best for a domain.

Representative evidence spans several roles:

Scroll sideways for more →

The table also shows why the label "VAE application" needs care. In some rows the model performs probabilistic inference with a continuous latent and ELBO. In others, a VAE-derived vector-quantized or perceptual autoencoder is a compression front end for a different generator. Both are historically connected, but their objectives, uncertainty interpretation, and sampling paths are not interchangeable.

Sequences, science, and recommendation

Bowman and colleagues trained a sentence VAE with a recurrent encoder and decoder, showing both the appeal of a global continuous sentence code and the posterior-collapse difficulty created by a powerful language decoder.^[7] MusicVAE used a hierarchical decoder that first generated subsequence embeddings and then generated musical subsequences, improving reconstruction, sampling, and interpolation over the paper's flat recurrent baseline.^[25]

For molecular design, Gómez-Bombarelli and colleagues encoded molecular strings into a continuous representation, added a property predictor, and optimized a surrogate objective in latent space before decoding candidate molecules. Their results established a concrete VAE-based design workflow, while also exposing validity problems when latent points decode to invalid strings.^[21]

Single-cell variational inference (scVI) used a hierarchical probabilistic model with neural conditional distributions for single-cell RNA sequencing. It represented each cell with latent normal variables while modeling count noise, library size, and batch annotations, supporting tasks such as batch correction, visualization, clustering, and differential expression in the reported experiments.^[22]

Mult-VAE adapted the VAE framework to collaborative filtering with implicit feedback and a multinomial likelihood. Its authors treated the KL coefficient as a task-specific regularization parameter and evaluated ranking rather than claiming that ancestral generation was the primary goal.^[23] This is an example of using variational representation learning inside a predictive system rather than using the model mainly to synthesize observations.

In the World Models system, a convolutional VAE compressed each game frame into a latent vector. A separate recurrent mixture-density model predicted latent dynamics, and a small controller consumed the latent and recurrent state.^[24] This work is an influential example of VAE-compressed observations in model-based control, but it should not be generalized to all later world models, which use several different latent-state constructions.

Audio and discrete codecs

Continuous sequence VAEs and vector-quantized codecs occupy related but distinct parts of audio modeling. MusicVAE is an actual hierarchical VAE over musical sequences.^[25] SoundStream instead uses a convolutional encoder-decoder and residual vector quantizer trained with reconstruction and adversarial losses to compress speech, music, and general audio.^[26] SoundStream belongs to the broader VQ-style codec lineage; calling every such codec a VAE would erase an important objective-level distinction.

Image tokenization and latent generation

Latent diffusion made autoencoders a prominent first stage for image generation. Rombach and colleagues trained perceptual compression models, then trained diffusion models in the lower-dimensional representation. They evaluated both a mild KL-regularized continuous latent and a vector-quantized alternative, showing a tradeoff between computational reduction and information lost through compression.^[19]

The autoencoder is a real quality boundary in this design. If the first stage discards information, the downstream generative model cannot reconstruct it. The rectified-flow system reported by Esser and colleagues continued to use a pretrained image autoencoder and found that increasing its latent channels from 4 to 16 improved reconstruction measures in their study, at the cost of a larger latent space for the generative model.^[20]

Research through the cutoff shows that this component is still evolving. SoftVQ-VAE proposed a differentiable continuous tokenizer that aggregates multiple codewords through soft categorical posteriors and reported image generation with 32 or 64 one-dimensional tokens.^[28] Work on video VAEs in 2025 studied joint spatial and temporal compression, keyframe branches, and group causal convolution for latent video diffusion.^[29] These papers support continued research on VAE-derived compression and tokenization; they do not support the stronger claim that one VAE architecture underlies every image or video generator.

Relationship to diffusion

There are two separate connections:

1. A diffusion probabilistic model can itself be expressed as a deep hierarchical latent-variable model optimized with a variational bound.^[27]
2. A latent diffusion or latent flow model can use a separately trained autoencoder to map pixels into a smaller space before iterative generation.^[19][20]

Conflating them leads to confusion. The first is a mathematical view of the diffusion process. The second is a modular system architecture. In the modular case, the first-stage model may be KL-regularized, vector-quantized, adversarially trained, or built from a newer tokenizer objective.

Choosing a VAE design

Latent structure

A single vector is convenient for small examples and global representations. Spatial feature maps preserve locality for images and are useful when a downstream model is convolutional. Multiple stochastic groups can represent information at different resolutions, as in hierarchical image VAEs, but they complicate sampling, KL accounting, and collapse diagnosis.^[6][17]

Latent dimension is only a capacity ceiling, not a measurement of information actually used. A 512-dimensional posterior with most dimensions close to the prior may carry less information than a smaller posterior with active coordinates. Per-dimension or per-group KL diagnostics are more informative than dimensionality alone.

Continuous or discrete latents

Continuous Gaussian latents provide direct pathwise gradients and a simple analytic KL. They support local interpolation, but the semantics of that interpolation are learned rather than guaranteed. Discrete codebooks provide finite symbols that can be modeled by transformers or other priors. They require quantization machinery, codebook utilization checks, and a separate definition of how the prior is learned.^[1][15]

The choice is not a simple quality ranking. DALL-E used a discrete image tokenizer because it later modeled a sequence of image and text tokens.^[18] Latent diffusion evaluated both KL-regularized and vector-quantized first stages while retaining two-dimensional spatial structure.^[19] A scientific model such as scVI instead benefits from a continuous probabilistic state and an observation distribution tailored to counts.^[22]

Decoder capacity

A weak decoder may force the latent variable to retain information but underfit the observation distribution. A highly expressive decoder may improve conditional likelihood while learning to ignore $z$. Sequence models make this conflict especially visible because an autoregressive decoder can predict a token from preceding tokens. Hierarchical decoding, limited receptive fields, latent injection at multiple layers, or adjusted training schedules can change the balance.^[5][7][25]

Decoder capacity should therefore be chosen with the goal in mind. If the latent is intended for a downstream task, information use must be measured. If marginal likelihood is primary, deliberately weakening the decoder may be counterproductive. If the decoder is a compression stage for another generator, reconstruction bandwidth and downstream cost become central.^[3][19][20]

Practical implementation and evaluation

Match the likelihood to the data

The decoder likelihood determines the reconstruction term and its scale. A Bernoulli likelihood is coherent for binary observations. A categorical likelihood is natural for discrete symbols. A Gaussian requires a decision about whether variance is fixed or learned. Count applications may need Poisson or negative-binomial structure; scVI's explicit count model is an example.^[3][22]

Using mean squared error without stating the implied Gaussian model hides assumptions about variance and relative weighting. Similarly, comparing ELBO values across implementations can be meaningless when preprocessing, dequantization, likelihood constants, or KL scaling differ.

Monitor more than total loss

Useful diagnostics include:

• expected log-likelihood and KL reported separately;
• KL per latent group or dimension;
• reconstructions from posterior samples and from posterior means;
• unconditional samples from the prior;
• the number of latent dimensions or groups carrying appreciable information;
• task-specific measures, such as held-out likelihood estimates, ranking metrics, molecular validity, or batch-correction diagnostics.

A low reconstruction error with nearly zero KL can indicate collapse. A large KL is not automatically better, because it may reflect poor prior matching or an overly permissive bottleneck. Prior samples and posterior reconstructions answer different questions and should both be inspected.^[3][5]

Treat modified objectives explicitly

If the KL coefficient is $\beta\neq1$, the objective is no longer the standard ELBO for the stated model without an additional interpretation. That can be an intentional rate-distortion or task-regularization choice, as in beta-VAE and Mult-VAE, but reports should name it rather than presenting every weighted sum as an unchanged likelihood bound.^[12][23]

For likelihood evaluation, an importance-weighted estimate can be tighter than a one-sample ELBO, although it remains an estimator with computational and statistical tradeoffs.^[8] For representation use, downstream validation is necessary. Smooth interpolations or visually orderly traversals are useful qualitative checks, but they do not establish identifiability or utility on their own.^[14]

Minimal implementation pattern

For a diagonal-Gaussian posterior, a framework-independent loss can be expressed as:

mu, logvar = encoder(x)
std = exp(0.5 * logvar)
epsilon = sample_standard_normal(shape(mu))
z = mu + std * epsilon
likelihood_parameters = decoder(z)

log_px_given_z = observation_log_prob(
    x,
    likelihood_parameters
).sum_over_observation_dimensions()

kl = -0.5 * (
    1 + logvar - square(mu) - exp(logvar)
).sum_over_latent_dimensions()

negative_elbo = mean(kl - log_px_given_z)

The observation log probability should come from the declared likelihood. If a library loss returns an average per pixel while KL is a sum per observation, the relative weighting changes with resolution. Before adding annealing or a custom coefficient, a reliable implementation check is to verify tensor shapes, reduction dimensions, signs, and the analytic KL against a trusted distribution library on small random inputs.

The stochastic path belongs in training. For a reconstruction diagnostic, both a posterior sample and the posterior mean can be decoded. For prior generation, the encoder is not used. Confusing these paths can make a deterministic reconstruction look like an unconditional sample or can accidentally evaluate only examples already seen by the encoder.

Stabilize the intended model, not just the scalar loss

Warm-up schedules, additional encoder updates, learned priors, posterior flows, hierarchical groups, and decoder changes intervene in different failure modes.^{[5][6][9][10]} Selecting one requires evidence about what is failing. For example, extra inference updates target a lagging encoder, while a flow targets posterior-family restriction and a learned prior targets mismatch between the aggregate posterior and a fixed prior.

Recent theory has begun to characterize optimization more formally. A 2025 analysis derived non-asymptotic convergence guarantees for specified VAE settings trained with stochastic gradient methods and Adam, including explicit dependence on assumptions and hyperparameters.^[30] Such results improve theoretical understanding, but they are not blanket guarantees for arbitrary architectures, data, or engineering choices.

Limitations and legacy

The standard VAE has several recurring limitations:

Scroll sideways for more →

VAEs remain important for three reasons. First, they made amortized variational inference and pathwise gradient estimation a standard neural modeling toolkit.^[1][2][4] Second, they provide a clear framework in which likelihood, compression, inference, and generation can be analyzed together.^[3] Third, their descendants and neighboring autoencoder lineages continue to supply learned representations for scientific models, recommendation, control, tokenization, and latent image or video generation.^{[19][21][22][23][24][28][29]}

Their legacy is therefore broader than the visual style of samples from an early Gaussian decoder. The durable contribution is a way to jointly learn a probabilistic generative model and fast approximate inference, plus a set of questions that remain active: how much information the latent variable should carry, how its distribution should be constrained, how the observation model should be chosen, and how compression interacts with a downstream generator.

References

1. ^Diederik P. Kingma and Max Welling, "Auto-Encoding Variational Bayes" (2013/2014). arxiv.org/...1312.6114
2. ^Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra, "Stochastic Backpropagation and Approximate Inference in Deep Generative Models" (2014). proceedings.mlr.press/...rezende14
3. ^Diederik P. Kingma and Max Welling, "An Introduction to Variational Autoencoders" (2019). arxiv.org/...1906.02691
4. ^Shakir Mohamed, Mihaela Rosca, Michael Figurnov, and Andriy Mnih, "Monte Carlo Gradient Estimation in Machine Learning" (2020). jmlr.org/...19-346
5. ^Junxian He, Daniel Spokoyny, Graham Neubig, and Taylor Berg-Kirkpatrick, "Lagging Inference Networks and Posterior Collapse in Variational Autoencoders" (2019). arxiv.org/...1901.05534
6. ^Casper Kaae Sonderby, Tapani Raiko, Lars Maaloe, Soren Kaae Sonderby, and Ole Winther, "Ladder Variational Autoencoders" (2016). papers.nips.cc/...3ec3b7c814df797cbda0f87-Abstract
7. ^Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew Dai, Rafal Jozefowicz, and Samy Bengio, "Generating Sentences from a Continuous Space" (2016). aclanthology.org/K16-1002
8. ^Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov, "Importance Weighted Autoencoders" (2015/2016). arxiv.org/...1509.00519
9. ^Danilo Rezende and Shakir Mohamed, "Variational Inference with Normalizing Flows" (2015). proceedings.mlr.press/...rezende15
10. ^Jakub M. Tomczak and Max Welling, "VAE with a VampPrior" (2018). proceedings.mlr.press/...tomczak18a
11. ^Kihyuk Sohn, Xinchen Yan, and Honglak Lee, "Learning Structured Output Representation using Deep Conditional Generative Models" (2015). papers.nips.cc/...6baa5c06772297520da2051-Abstract
12. ^Irina Higgins et al., "beta-VAE: Learning Basic Visual Concepts with a Constrained Variational Framework" (2017). openreview.net/forum
13. ^Ricky T. Q. Chen, Xuechen Li, Roger Grosse, and David Duvenaud, "Isolating Sources of Disentanglement in Variational Autoencoders" (2018). proceedings.neurips.cc/...a35977997223d22-Abstract
14. ^Francesco Locatello et al., "Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations" (2019). proceedings.mlr.press/...locatello19a
15. ^Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu, "Neural Discrete Representation Learning" (2017). papers.nips.cc/...63a0ac09ce2e96d03992fbc-Abstract
16. ^Ali Razavi, Aaron van den Oord, and Oriol Vinyals, "Generating Diverse High-Fidelity Images with VQ-VAE-2" (2019). papers.nips.cc/...18d1bbcadf1cd9c7a54fb8c-Abstract
17. ^Arash Vahdat and Jan Kautz, "NVAE: A Deep Hierarchical Variational Autoencoder" (2020). papers.nips.cc/...83cf7c2c7a66be163579d37-Abstract
18. ^Aditya Ramesh et al., "Zero-Shot Text-to-Image Generation" (2021). proceedings.mlr.press/...ramesh21a
19. ^Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjorn Ommer, "High-Resolution Image Synthesis with Latent Diffusion Models" (2022). openaccess.thecvf.com/...on_Models_CVPR_2022_paper
20. ^Patrick Esser et al., "Scaling Rectified Flow Transformers for High-Resolution Image Synthesis" (2024). proceedings.mlr.press/...esser24a
21. ^Rafael Gomez-Bombarelli et al., "Automatic Chemical Design Using a Data-Driven Continuous Representation of Molecules" (2018). pmc.ncbi.nlm.nih.gov/...PMC5833007
22. ^Romain Lopez, Jeffrey Regier, Michael B. Cole, Michael I. Jordan, and Nir Yosef, "Deep Generative Modeling for Single-cell Transcriptomics" (2018). pmc.ncbi.nlm.nih.gov/...PMC6289068
23. ^Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffman, and Tony Jebara, "Variational Autoencoders for Collaborative Filtering" (2018). arxiv.org/...1802.05814
24. ^David Ha and Jurgen Schmidhuber, "World Models" (2018). arxiv.org/...1803.10122
25. ^Adam Roberts, Jesse Engel, Colin Raffel, Curtis Hawthorne, and Douglas Eck, "A Hierarchical Latent Vector Model for Learning Long-Term Structure in Music" (2018). research.google/...ng-long-term-structure-in-music
26. ^Neil Zeghidour, Alejandro Luebs, Ahmed Omran, Jan Skoglund, and Marco Tagliasacchi, "SoundStream: An End-to-End Neural Audio Codec" (2021). research.google/...n-end-to-end-neural-audio-codec
27. ^Diederik P. Kingma, Tim Salimans, Ben Poole, and Jonathan Ho, "Variational Diffusion Models" (2021). papers.nips.cc/...a0229873fefc2a4b06377fa-Abstract
28. ^Hao Chen et al., "SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer" (2025). openaccess.thecvf.com/...Tokenizer_CVPR_2025_paper
29. ^Pingyu Wu, Kai Zhu, Yu Liu, Liming Zhao, Wei Zhai, Yang Cao, and Zheng-Jun Zha, "Improved Video VAE for Latent Video Diffusion Model" (2025). openaccess.thecvf.com/...ion_Model_CVPR_2025_paper
30. ^Sobihan Surendran, Antoine Godichon-Baggioni, and Sylvain Le Corff, "Theoretical Convergence Guarantees for Variational Autoencoders" (2025). proceedings.mlr.press/...surendran25a