overfitting
Cross-validation: how you find out whether a model learned anything or just memorised the answers Lesson
A model's score on the data it trained on tells you nothing about whether it will work, because memorising is easier than learning. Cross-validation and holdout sets solve this by measuring the model only on examples it has never seen, and the discipline of keeping a final test set untouched is what separates a real result from a self-flattering one.
The bias-variance tradeoff: why a model can fail by being too simple or too clever Lesson
Every model's error splits into two opposing parts -- bias, from being too rigid to capture the pattern, and variance, from being so flexible it memorizes noise. Reducing one usually raises the other, and the whole craft of machine learning is finding where their sum is smallest.
Regularization: why deliberately handicapping a model makes it better Lesson
Regularization is the family of techniques that make a model perform worse on its training data in order to make it perform better on data it has never seen, with dropout and weight decay the two that shaped modern deep learning.