hyperparameters

Learning rate schedules and warmup Lesson

A learning rate schedule changes how big each training step is over the course of a run, and warmup is the near-universal practice of starting tiny and ramping up -- without it, large transformer training runs frequently diverge in the first few hundred steps.