Learn · Intermediate

Batch normalization: grading every layer on a curve so deep networks train faster

Batch normalization is a layer that standardizes the numbers flowing through a neural network, using the average and spread of the current training batch, and then lets the network learn whatever scale and offset it actually prefers. Introduced by Sergey Ioffe and Christian Szegedy at Google in 2015, it made deep image networks train dramatically faster and more reliably, and it became a default ingredient of computer-vision models for years afterwards.

The original paper, Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift, reported that on a leading image classifier of the day, batch normalization reached the same accuracy "with 14 times fewer training steps." It also allowed "much higher learning rates" and less careful setup. Few single changes to a network have ever bought that much.

The problem it set out to fix

A deep network is a stack of layers, each transforming the output of the one below. During training, gradient descent nudges every layer's weights after every batch. That means each layer is trying to learn from inputs whose typical size and centre keep drifting as the layers beneath it change. Ioffe and Szegedy called this drift "internal covariate shift."

The practical symptom was fragility. Values could grow very large or shrink towards zero as they passed through many layers, pushing activation functions into flat regions where the gradient barely moves. Engineers compensated with small learning rates and carefully tuned initial weights, which made training slow and finicky.

How it works

Think of grading on a curve. A teacher takes each class's raw exam scores, subtracts the class average and divides by the spread, so every class ends up on the same scale. Then, because a pure curve might not suit the subject, the school is allowed to stretch and shift the curved scores however it wants.

Batch normalization does exactly that, one feature at a time:

1. During training, for each feature, compute the mean and variance across the examples in the current mini-batch. 2. Subtract the mean and divide by the standard deviation (plus a tiny constant so nothing divides by zero). The feature is now centred near zero with a spread near one. 3. Multiply by a learned scale and add a learned shift. These two parameters are trained like any other weight.

Step 3 is what makes the method safe. If the network decides a feature works better with a different centre or spread, it can learn to undo the normalization entirely. Batch normalization does not force a layout on the network. It gives every layer a predictable starting point.

Training mode versus inference mode

At inference time there is often no batch to average over: a phone app classifies one photo at a time. So while training, each batch-norm layer keeps a running average of the means and variances it has seen, and at inference it uses those stored values instead. That split, discussed more generally in training vs inference, is the source of a classic bug. A model accidentally left in training mode computes statistics from whatever happens to share its batch, so the same input can give different outputs depending on its neighbours.

The same batch dependence has an upside. Because each example's normalized value depends slightly on which other examples were drawn with it, training gets a little noise, which acts as a mild regularizer. The original paper noted that it could sometimes remove the need for dropout; our lesson on regularization covers why noise of that kind helps.

Why it works is still argued over

The original explanation did not hold up well. In 2018, Shibani Santurkar and colleagues at MIT published How Does Batch Normalization Help Optimization?, which found that the "distributional stability of layer inputs has little to do with the success of BatchNorm." Their alternative: batch normalization "makes the optimization landscape significantly smoother," so gradients become more predictable and larger steps become safe. The technique survived; the story about it changed. That is a useful reminder that a method can work reliably while its authors' account of why remains wrong.

Where it breaks, and what replaced it

Batch normalization needs a batch big enough to give honest statistics. Yuxin Wu and Kaiming He, in Group Normalization (2018), documented that its "error increases rapidly when the batch size becomes smaller," which is exactly the regime of large models, high-resolution detection and video, where memory allows only a few examples per chip. Their fix, group normalization, computes statistics within each example instead of across the batch.

Language models went further in the same direction. Transformers almost universally use layer normalization, from Jimmy Lei Ba, Jamie Ryan Kiros and Geoffrey Hinton's 2016 paper, which normalizes across the features of a single example. Text arrives in batches of uneven length, and a model generating one word at a time has no meaningful batch to consult, so a per-example method is simply a better fit.

Batch normalization remains common in convolutional networks, and its core lesson outlived its dominance: deep networks train well when the numbers moving between layers are kept on a predictable scale. Layer normalization, residual connections and careful learning-rate warmup are all, in different ways, answers to the same problem.

Ground Truth. (2026, September 11). Batch normalization: grading every layer on a curve so deep networks train faster. Ground Truth. https://groundtruth.day/learn/batch-normalization.html

@misc{groundtruth:batch-normalization,
  title  = {Batch normalization: grading every layer on a curve so deep networks train faster},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/learn/batch-normalization.html}
}

Topics: fundamentals · training · optimization · normalization · computer-vision · deep-learning