Learn · Intermediate

Layer normalization: the rescaling that keeps deep networks trainable

Layer normalization takes the numbers flowing through each layer of a neural network and rescales them to a consistent range: subtract the average, divide by the spread. That one step is what keeps signals from exploding or vanishing as they pass through dozens of stacked layers. Along with residual connections, it is the reason a deep transformer can be trained at all, and it appears in essentially every large language model shipped today.

The technique comes from a 2016 paper by Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton, written to fix a specific weakness in the technique that preceded it.

The problem

A neural network is a stack of layers, each transforming the numbers handed to it and passing the result on. Each layer multiplies its input by learned weights and applies some function. Nothing constrains the scale of what comes out.

That is fine for two layers and catastrophic for fifty. If each layer amplifies its input slightly -- say by 1.1 -- then after fifty layers the signal is 117 times larger. If each shrinks it by 0.9, the signal is down to five thousandths of what it was. This is the exploding and vanishing gradient problem, and it is why deep networks were mostly a theoretical curiosity for decades. The math said depth should help. In practice, training just collapsed.

The imagined fix is obvious: keep the scale under control. The question is how.

Batch normalization, introduced in 2015, did it by normalizing each feature across all the examples in a training batch. It worked spectacularly for image networks, and it has a strange property: an example's output depends on which other examples happened to be in its batch. For images that is tolerable. For text it is a problem, because sentences have different lengths, batches are often small, and at inference time you might be processing exactly one thing.

The fix

Layer norm makes one change of perspective. Instead of normalizing each feature across the batch, it normalizes across the features within a single example.

Concretely: take the vector representing one token at one layer -- a list of, say, 4,096 numbers. Compute their mean. Compute their standard deviation. Subtract the mean from each, divide each by the standard deviation. Now the vector has mean 0 and standard deviation 1, guaranteed, regardless of what came in. Then multiply by a learned scale and add a learned offset, so the network can recover a different range if it needs one.

The crucial part: this calculation involves only that one example. No batch, no neighbors, no dependence on what else is being processed. Feed the model one token or ten thousand and layer norm behaves identically.

The analogy: batch norm is grading a test on a curve -- your score depends on how everyone else did, so the same paper gets different marks in different classrooms. Layer norm is grading each paper against itself: what were this student's strongest and weakest answers, and how do the rest compare? The result depends on nothing but the paper in front of you.

Where it sits, and why that matters

The original transformer paper placed layer norm after each sublayer's residual addition -- Post-LN. It worked, but training deep transformers this way is notoriously delicate, requiring a warmup schedule where the learning rate creeps up from near zero, or the whole thing diverges in the first few hundred steps.

A 2020 paper worked out why and moved it before each sublayer instead -- Pre-LN. This keeps the residual pathway clean, giving gradients an unobstructed route from the top of the network to the bottom. Pre-LN trains stably without warmup, and it is what essentially every modern large model uses. A change of a few lines, and it made frontier-scale training routine.

The other refinement in wide use is RMSNorm, from 2019, which drops the mean-subtraction step and only rescales by the root-mean-square magnitude. The finding was that recentering contributed little -- almost all the benefit came from controlling scale. Removing it is cheaper, and it works just as well.

The intuition worth keeping

Layer norm does not make the network smarter. It carries no information and learns no pattern. What it does is keep the numbers in a range where the rest of the machinery -- gradient descent, the optimizer, the activation functions -- can do its job.

That is the recurring shape of deep learning's most important ideas. The famous ones are about capability. The ones that actually unlocked scale -- residual connections, normalization, careful initialization -- are about keeping a very tall stack of arithmetic numerically sane long enough for the learning to happen. They are the scaffolding, not the building, and nothing gets built without them.

Ground Truth. (2026, July 14). Layer normalization: the rescaling that keeps deep networks trainable. Ground Truth. https://groundtruth.day/learn/layer-normalization.html

@misc{groundtruth:layer-normalization,
  title  = {Layer normalization: the rescaling that keeps deep networks trainable},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/layer-normalization.html}
}

Topics: fundamentals · transformers · training · architecture · normalization