Learn · Intermediate

Vector Quantization: Turning Continuous Data Into a Vocabulary

Vector quantization is the trick that gives images, audio and video a vocabulary. It takes the continuous vectors a neural network produces and snaps each one to the nearest entry in a learned dictionary of reference vectors, so a picture becomes a grid of dictionary indices - a sequence of symbols. Once your data is symbols, every tool built for predicting the next word in a sentence works on it unchanged. Almost every discrete image tokenizer, neural audio codec and token-based video model rests on this idea.

The motivation comes from a mismatch. Language arrives pre-chopped into a finite set of pieces; tokenization just formalises a division that already exists. Images do not work that way. A photograph is millions of continuous numbers with no natural units, and "predict the next real number" is a badly behaved problem - there is no cleanest answer, only a distribution over infinitely many.

Aaron van den Oord and colleagues at DeepMind published the standard answer in 2017 as VQ-VAE, or the Vector Quantized Variational Autoencoder. The architecture has three parts. An encoder compresses the input into a grid of continuous vectors. A quantizer replaces each vector with the closest entry in a codebook - say 512 or 8,192 learned reference vectors - and records only that entry's index. A decoder reconstructs the original from the indices alone.

The useful analogy is a paint-by-numbers kit. A photograph contains millions of distinct colours, but you can approximate it well with a palette of 64 numbered paints, and then the picture is fully described by a grid of numbers between 1 and 64. The codebook is the palette. The crucial difference is that the network learns which 64 colours to include, choosing them so that reconstructions come out as faithful as possible.

One obstacle has to be handled explicitly. Snapping to the nearest codebook entry is a hard, non-differentiable choice, so gradients cannot flow back through it during backpropagation. VQ-VAE uses a straight-through estimator: in the forward pass the quantized vector is used, and in the backward pass the gradient is simply copied past the snapping step as if it had not happened. It is a deliberate approximation that works well in practice. Two extra loss terms keep the system honest - one pulls codebook entries toward the encoder outputs assigned to them, and a commitment term pulls encoder outputs toward their chosen entries so the encoder does not wander away from the palette.

The famous failure mode is codebook collapse. If most entries are never selected, they receive no gradient, drift nowhere useful, and stay unused forever - so a nominal vocabulary of 8,192 does the work of 200. Practitioners fight this with codebook resets, exponential moving average updates, and lower-dimensional codes. Fabian Mentzer and colleagues at Google Research proposed a blunter fix in 2023 with finite scalar quantization, which throws away the learned codebook entirely: project down to a handful of dimensions, round each one to a small fixed set of levels, and let the implied grid of combinations be the vocabulary. No codebook, no collapse, and comparable quality. That simplicity is why newer systems keep choosing it.

What this unlocked is best seen in what got built on top. Patrick Esser and colleagues combined a vector-quantized image tokenizer with a transformer in VQGAN, making high-resolution image generation an autoregressive next-token problem. Neural audio codecs like SoundStream, from Neil Zeghidour and colleagues at Google, use residual vector quantization - a stack of codebooks where each one encodes what the previous stage got wrong - to compress speech and music into token streams, which is how modern text-to-speech systems generate audio the same way a language model generates text.

The idea keeps resurfacing in new places. The PhiZero paper covered in today's roundup of world-model papers applies finite scalar quantization not to pixels but to what changes between frames, producing a small alphabet of discrete transition symbols that a vision-language model predicts and a diffusion decoder renders. The authors are careful to say this alphabet is an empirical description of state transitions, not a discovered language of physics - a useful reminder that quantization compresses whatever you point it at, and gives it no meaning it did not already have.

The honest limitations are worth carrying. Quantization always discards information, and the discarded part is disproportionately fine detail and rare cases - exactly what model-collapse concerns are about. Codebook size is a genuine trade-off: too small and reconstructions blur, too large and the vocabulary is under-trained and hard to predict. And a discrete code that reconstructs beautifully can still be a poor thing to predict, because reconstruction quality and predictability are different objectives. That gap is where most of the current research sits.

If you want the intuition in one sentence: vector quantization is the decision to stop describing the world exactly and start describing it with a limited set of words, on the bet that a limited set of words is what makes prediction possible at all. Related reading: embeddings, tokenization, and how AI picks its next word.

Ground Truth. (2026, August 1). Vector Quantization: Turning Continuous Data Into a Vocabulary. Ground Truth. https://groundtruth.day/learn/vector-quantization-and-discrete-latent-codes.html

@misc{groundtruth:vector-quantization-and-discrete-latent-codes,
  title  = {Vector Quantization: Turning Continuous Data Into a Vocabulary},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/vector-quantization-and-discrete-latent-codes.html}
}

Topics: representation-learning · tokenization · generative-models · world-models · audio · video