optimization
Curriculum learning: why the order of the examples matters Lesson
Training a model on easy examples before hard ones can make it learn faster and end up better than showing the same data in random order, which is why frontier labs now build ladders of increasingly difficult tasks.
Learning rate schedules and warmup Lesson
A learning rate schedule changes how big each training step is over the course of a run, and warmup is the near-universal practice of starting tiny and ramping up -- without it, large transformer training runs frequently diverge in the first few hundred steps.
Loss of plasticity: why a network that trains forever stops being able to learn Lesson
Keep training a neural network on a stream of new tasks and it does not just forget the old ones - it gradually loses the ability to learn new ones at all, until it performs worse than a freshly initialized network.
What Is Gradient Descent? Lesson
Gradient descent is the optimization method that trains almost every modern neural network: it repeatedly nudges the model's parameters a small step in the direction that most reduces its error, until the error stops falling.