Learn · Intermediate

Forward and Reverse KL Divergence

KL divergence measures how far one probability distribution is from another, in units of wasted information. Its defining property is that it is asymmetric: the distance from distribution P to distribution Q is not the same as from Q to P. That asymmetry is not a mathematical wart - it is the single most consequential design choice in distillation, variational inference, and reinforcement learning from feedback, because the two directions produce models that fail in opposite ways.

The quantity was introduced by Solomon Kullback and Richard Leibler in a 1951 paper in the Annals of Mathematical Statistics, and the information-theoretic reading is the one worth carrying around. Suppose the true distribution of some data is P, but you build your compression scheme assuming it is Q. You will need more bits than necessary, because your code assigns short symbols to things that are actually rare. The KL divergence from P to Q is exactly that average waste. If Q equals P, you waste nothing and the divergence is zero. It is never negative, and - crucially - it is not a distance, because it fails the symmetry a distance requires.

Concretely, it is the expected log-ratio of the two probabilities, averaged under whichever distribution you write first. That "whichever you write first" is where everything happens.

Forward KL: cover everything

Forward KL averages under the target, P. Every outcome that P considers possible contributes to the loss, weighted by how likely P thinks it is. If your model Q assigns near-zero probability to something P produces regularly, the log-ratio explodes and the penalty is enormous.

So forward KL is mode-covering. It cannot tolerate a model that says "impossible" about something real. The model is forced to smear probability mass over the entire support of the target, even into regions it cannot represent well.

This is the objective you are already using without knowing it. Standard next-token training on a fixed corpus - maximum likelihood, cross-entropy loss - is forward KL from the data distribution to the model. It is also what Geoffrey Hinton, Oriol Vinyals and Jeff Dean used in the paper that introduced neural distillation: train the student to match the teacher's full softened output distribution.

Reverse KL: commit to something

Reverse KL averages under the model, Q. Now only the outcomes your model actually produces contribute. Regions where the target has probability but your model does not are simply never sampled, so they cost nothing. But if your model puts mass somewhere the target considers unlikely, the ratio blows up and you are punished hard.

So reverse KL is mode-seeking. It says: whatever you choose to do, do something the target would endorse. It has no opinion about what you leave out. Give a model a target with three plausible answers and reverse KL is perfectly happy for it to pick one and always give that one.

Reverse KL is the objective behind variational inference, including the ELBO in Kingma and Welling's variational autoencoder, and it is the KL term in the RLHF objective that keeps a fine-tuned policy from drifting too far from its base model.

The analogy

Imagine forecasting a friend's dinner order. They order Thai, Italian or sushi, roughly evenly.

Under forward KL, you must never say a real choice is impossible, so you predict a blur across all three - a forecast that describes something they would never actually order on any given night. Under reverse KL you may confidently say "Thai," which is wrong two nights in three but is never absurd: every prediction you make is a thing they genuinely do.

Forward KL gives you an averaged answer that satisfies no case. Reverse KL gives you a specific answer that ignores real alternatives. Neither is correct in general. The right choice depends on what a mistake costs.

Why this matters right now

When a student model has roughly the capacity of its teacher, forward KL is fine - it can afford to cover everything. When the student is dramatically smaller, forward KL is actively harmful: forced to hedge across a distribution it cannot represent, it produces vague, averaged, low-quality text. This is why modern LLM distillation moved to reverse KL, following work such as MiniLLM and Google's on-policy distillation, which additionally evaluates the teacher on sequences the student generated rather than on a fixed dataset.

That combination - reverse KL, on the student's own trajectory - is now the standard recipe, and it turned up in three separate papers in a single day in July 2026: one distilling a search protocol into an open model, one distilling planning behavior token-by-token during the student's own rollout, and one distilling guided diffusion. All three minimize reverse KL, and all three do it on-policy.

The honest caveats. Reverse KL's mode-seeking behavior is mode collapse when you did not want it - a chat model that always answers the same way, an image model that generates one face. Forward KL requires knowing the target's probabilities everywhere, which you often cannot get from a commercial API that returns text rather than distributions. And both directions are undefined when the target assigns exactly zero probability to something the model produces, which is why implementations clamp, smooth, or mix the two.

There are symmetric alternatives - Jensen-Shannon divergence, which averages both directions, and generalized families that interpolate between them - and they are used when neither failure mode is acceptable. But they are compromises, not free lunches.

The takeaway: whenever you see a KL term in a loss function, check the order of the arguments before anything else. It tells you whether the model was built to hedge or to commit.

Related lessons: distillation, softmax and cross-entropy, perplexity, on-policy vs off-policy learning, and RL post-training.

Ground Truth. (2026, July 28). Forward and Reverse KL Divergence. Ground Truth. https://groundtruth.day/learn/forward-and-reverse-kl-divergence.html

@misc{groundtruth:forward-and-reverse-kl-divergence,
  title  = {Forward and Reverse KL Divergence},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/forward-and-reverse-kl-divergence.html}
}

Topics: fundamentals · training · distillation · probability · loss-functions