Learn · Intermediate

Learning rate schedules and warmup

A learning rate schedule is the rule that decides how large each weight update is at each point during training, and warmup is the standard opening move: start the learning rate near zero and ramp it up over the first few hundred or few thousand steps before decaying it back down. This is not a minor tuning detail. Remove warmup from a large transformer training run and it will often diverge outright in the first few hundred steps, turning weeks of compute into a wall of not-a-number errors.

The learning rate is the single most consequential number in gradient descent. Each training step computes a direction to move the weights and then moves some distance in that direction. The learning rate is that distance. Too small and training crawls; too large and each step overshoots, the loss climbs instead of falling, and the run blows up. What makes this hard is that the right value is not constant. The best step size at step 10 is very different from the best step size at step 100,000.

The shape almost everyone uses

The canonical modern schedule has three parts. First, warmup: the learning rate rises linearly from roughly zero to its peak over some number of steps. Then the peak. Then decay: the rate falls smoothly, usually following a cosine curve, down to a small fraction of the peak by the final step.

The warmup half of this comes from Attention Is All You Need, the 2017 paper from Ashish Vaswani and colleagues at Google that introduced the transformer. Their schedule increased the learning rate linearly for the first 4,000 steps and then decayed it in proportion to the inverse square root of the step number. That 4,000-step warmup was presented almost in passing, and it has been in essentially every large training recipe since.

The decay half traces to work on cyclical and cosine schedules -- Leslie Smith's Cyclical Learning Rates and Ilya Loshchilov and Frank Hutter's SGDR, which introduced the cosine shape that later became the default for language model pretraining.

Why warmup is necessary, not just helpful

There are two good explanations and they are both partly true.

The first is about the optimizer. Modern training uses adaptive optimizers like Adam, which scale each parameter's step by running estimates of the gradient's recent average and variance. At step one, those estimates are built from a single noisy batch. Dividing by a variance estimate computed from almost no data produces wild step sizes for some parameters. Warmup keeps the actual steps small during exactly the window when the optimizer's own statistics are untrustworthy. The analogy is a new driver: the problem is not that they cannot steer, it is that their sense of how much to steer has not calibrated yet, so you keep the speed down until it has.

The second is about architecture. On Layer Normalization in the Transformer Architecture, by Ruibin Xiong and colleagues, showed that where you place the normalization layer determines how badly you need warmup. In the original post-normalization design, gradients near the output layer are very large at initialization, and a full-size first step is destructive. Moving the normalization inside the residual branch -- pre-normalization, now the standard choice -- tames those gradients and makes training far more forgiving. This is one of the clearer cases where an architectural fix and a training-schedule fix address the same underlying problem, covered further in layer normalization.

The mistake that quietly costs you a model

A cosine schedule is defined relative to a total number of steps. It reaches its minimum exactly at the end. If you set the schedule for 500,000 steps and stop at 200,000, your model finishes training at a still-high learning rate and never gets the fine-grained settling that the tail of the schedule provides. The authors of Training Compute-Optimal Large Language Models -- the Chinchilla paper, from Jordan Hoffmann and colleagues at DeepMind -- flagged this explicitly, noting that a cosine cycle length mismatched to the actual training horizon produces a measurably worse model. Since that paper is best known for reshaping how the field thinks about scaling laws, it is easy to miss that one of its practical findings is simply: make your schedule end where your training ends.

In practice, the numbers you will see in real recipes are unglamorous and fairly stable. Warmup is commonly a few hundred to a few thousand steps, or a small percentage of total steps. Peak learning rates for large models are often in the range of one to three ten-thousandths, decaying to about a tenth of the peak. Fine-tuning uses much smaller values than pretraining, because you are adjusting a model that is already good rather than building one from noise -- see fine-tuning and LoRA. And when a training run diverges, the learning rate schedule is the first thing to check, before anything more interesting.

Ground Truth. (2026, August 10). Learning rate schedules and warmup. Ground Truth. https://groundtruth.day/learn/learning-rate-schedules-and-warmup.html

@misc{groundtruth:learning-rate-schedules-and-warmup,
  title  = {Learning rate schedules and warmup},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/learning-rate-schedules-and-warmup.html}
}

Topics: fundamentals · training · optimization · hyperparameters · transformers