perplexity

Perplexity: the number that tells you a model still works Lesson

Perplexity measures how surprised a language model is by real text, expressed as the number of words it was effectively choosing between at each step -- lower means less confused, and a jump from single digits into the thousands means the model is broken.