Learn · Beginner

Autoencoders and VAEs: teaching a model to compress the world

An autoencoder is a neural network trained to do something that sounds pointless: take an input, squeeze it through a deliberately narrow bottleneck, and reproduce the input on the other side. The point is the bottleneck. To rebuild a photograph from a few hundred numbers, the network has to discover what actually matters in photographs. A variational autoencoder, introduced by Diederik Kingma and Max Welling in 2013, adds one change that turned this into the foundation of modern generative AI: it makes the bottleneck a smooth space of probabilities rather than a scatter of isolated points, so you can sample new data from it.

The plain autoencoder

The architecture is two halves. An encoder maps the input down to a small vector, usually called the latent code or just the latent. A decoder maps that vector back up to something the same shape as the input. The training signal is reconstruction error: how far is the output from what went in. No labels are required, which makes this one of the original self-supervised methods.

Think of it as forced note-taking. If you must summarize a novel in one index card and later reconstruct the plot from that card alone, you will stop recording individual sentences and start recording structure: characters, arcs, the turning points. The constraint is what produces the abstraction. Make the bottleneck too wide and the network cheats by copying; make it too narrow and it loses things it needed. That tension is the whole design.

Those learned codes are useful on their own. They are close relatives of embeddings, and they get used for denoising, anomaly detection (things the model reconstructs badly are things it has never seen), and dimensionality reduction.

Why plain autoencoders cannot generate

Here is where it breaks. Suppose you train an autoencoder on faces, then invent a random latent vector and hand it to the decoder, hoping for a new face. You will almost certainly get garbage. Nothing in the training ever asked the latent space to be continuous. The encoder is free to put one face at coordinate 4.1 and a completely different one at 4.2, with nothing coherent between them. The space is a set of memorized addresses, not a map.

A VAE fixes this with two ideas. First, the encoder outputs a distribution instead of a point: a mean and a spread for each latent dimension. During training you sample from that distribution, so a single input maps to a small fuzzy cloud rather than a pinpoint, and nearby points in the cloud must all decode to something sensible. Second, the loss adds a term pulling every one of those clouds toward a standard bell curve centered at zero, measured with KL divergence. That term is a pressure toward a tidy, gap-free space, and it fights the reconstruction term, which wants each input parked far from every other for maximum fidelity. The balance between them is the VAE's central knob.

The result is a latent space you can actually walk through. Sample a random point from the bell curve, decode, and you get a plausible new example. Interpolate between two encoded inputs and you get a smooth morph rather than a jump cut. Kingma and Welling's paper also contributed the reparameterization trick, the technical move that makes this trainable at all: instead of sampling in a way that blocks gradients, you sample a fixed noise value and reshape it with the predicted mean and spread, so backpropagation can flow straight through the randomness. Rezende, Mohamed, and Wierstra derived the same approach independently the following year.

Why this still matters in 2026

VAEs lost the headline generative-quality contest to GANs and then to diffusion models, because their samples tend to be blurry. Averaging is the safe bet under reconstruction loss, and averages of sharp things are soft.

But the autoencoder came back through the side door, and it now sits underneath almost everything. When Robin Rombach and colleagues introduced latent diffusion in 2021, the architecture behind Stable Diffusion, the key move was to stop running diffusion on pixels. Instead, an autoencoder compresses the image into a much smaller latent grid, diffusion runs entirely in that compressed space, and the decoder converts the result back to pixels once at the end. That is why a consumer GPU can generate a high-resolution image at all. Video models do the same thing across time.

The discrete branch matters too. Aaron van den Oord's VQ-VAE swaps the continuous latent for entries in a learned codebook, turning an image into a sequence of discrete tokens, which is exactly the format a transformer already knows how to model. That lineage runs through vector quantization into today's image, audio, and speech tokenizers.

The honest caveat: the compression is lossy on purpose, and what it discards is a design decision nobody fully controls. Fine text, small faces, and hands have historically suffered because the autoencoder was never rewarded for keeping them. When a generated image looks right at a glance and wrong in the details, the bottleneck is often where the detail went.

Ground Truth. (2026, August 5). Autoencoders and VAEs: teaching a model to compress the world. Ground Truth. https://groundtruth.day/learn/autoencoders-and-variational-autoencoders.html

@misc{groundtruth:autoencoders-and-variational-autoencoders,
  title  = {Autoencoders and VAEs: teaching a model to compress the world},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/autoencoders-and-variational-autoencoders.html}
}

Topics: fundamentals · generative-models · representation-learning · vae · diffusion