probability

Forward and Reverse KL Divergence Lesson

KL divergence measures how far one probability distribution is from another, and it is asymmetric: swapping which distribution goes first changes whether your model tries to cover everything the target does or commit to one part of it.