Learn · Intermediate

Gradient clipping: the one-line fix that keeps big models from blowing up mid-training

Gradient clipping caps how large a single weight update can be during training. When a neural network computes how to correct itself after a batch of data, that correction signal, the gradient, occasionally comes back enormous, and applying it as-is would throw the model's weights into nonsense that hours or days of training cannot recover from. Clipping shortens the update while keeping its direction, so the model still learns the right lesson without overreacting to one bad example. It is a handful of lines of code, it is in essentially every large training run in production, and it is the reason those runs finish.

To see why the problem exists, you need to know what a gradient is. Training a network means repeatedly nudging its weights in whatever direction reduces error, a process called gradient descent. The gradient is the vector that says which way is downhill and how steep the slope is. Ordinarily it is modest, and the model takes a small, sensible step.

The trouble comes from the fact that gradients are computed by backpropagation, which works backwards through the network multiplying terms together layer by layer. Multiplication compounds. If each layer contributes a factor slightly above one, then across fifty layers the product is enormous; if each is slightly below one, the product vanishes to nothing. Those are the twin failures the field named exploding gradients and vanishing gradients, and Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio laid out both precisely in their 2012 paper On the difficulty of training Recurrent Neural Networks.

Their geometric picture is the one worth keeping. Imagine the error surface a model is descending as a landscape. Most of it is gently rolling, and steady small steps work fine. But the surface can contain a cliff, a place where the slope is suddenly near-vertical. A model walking along the flat part takes a normal-sized step, hits the cliff edge, and the gradient there is so steep that the step it computes flings it kilometres away, into terrain it has never seen and cannot get back from. Nothing about the model was broken. It just took one honest step in a place where the honest step was catastrophic.

Clipping puts a leash on the step. The common form is norm clipping: measure the total length of the gradient vector across all parameters, and if it exceeds a threshold, rescale the whole vector down to exactly that length. Direction preserved, magnitude capped. In PyTorch it is torch.nn.utils.clip_grad_norm_(model.parameters(), max_norm=1.0), called between computing gradients and applying them. A threshold of 1.0 is a common default in language model training.

The obvious question is why not simply lower the learning rate instead. The answer is that this treats a rare event by penalizing every ordinary one. Explosions happen on a small fraction of batches; the other 99% are fine and want full-speed steps. Shrinking the learning rate enough to survive the worst batch makes the other batches crawl. Clipping is targeted: it does nothing at all on a normal step and intervenes only when a step is about to be absurd. It is a circuit breaker, not a dimmer switch, which is also why it composes cleanly with learning rate schedules and warmup rather than competing with them.

There is also a theoretical result behind the practice. Jingzhao Zhang and colleagues argued in Why gradient clipping accelerates training that the standard analysis of gradient descent assumes the gradient's smoothness is bounded by a single global constant, which real neural network loss surfaces violate. Under a more realistic assumption where local smoothness varies with the gradient's own magnitude, clipped descent can converge faster than unclipped descent. So clipping is not only a safety measure bolted on after the fact. Under conditions that actually describe deep networks, it is the better algorithm.

Gradient clipping is one member of a family of stability tools, and it is worth knowing where it sits. Residual connections give gradients a shortcut path so they neither explode nor vanish as badly on the way back. Layer normalization keeps activations in a well-behaved range so the gradients derived from them stay reasonable. Mixed-precision training introduces its own version of the problem, since sixteen-bit numbers overflow far sooner than thirty-two-bit ones, which is why loss scaling exists alongside clipping. And the LSTM, introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997, was an architectural answer to the vanishing half of the problem, adding gated paths that let signal travel across many steps without being multiplied away.

Two practical notes. First, a threshold that is too aggressive is its own failure mode: clip hard enough and you throw away real signal, and the model learns slowly for reasons that look mysterious. Watching the fraction of steps that get clipped is more informative than watching the threshold. Second, when a training run diverges, clipping is the first thing to check and often the wrong thing to blame. A model that explodes even with clipping in place usually has a deeper problem, such as a bad initialization, a learning rate set far too high, or corrupted data, and the clipping is just the alarm that went off rather than the fire.

Ground Truth. (2026, September 2). Gradient clipping: the one-line fix that keeps big models from blowing up mid-training. Ground Truth. https://groundtruth.day/learn/gradient-clipping-and-exploding-gradients.html

@misc{groundtruth:gradient-clipping-and-exploding-gradients,
  title  = {Gradient clipping: the one-line fix that keeps big models from blowing up mid-training},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/learn/gradient-clipping-and-exploding-gradients.html}
}

Topics: training · optimization · gradients · stability · fundamentals