normalization

Layer normalization: the rescaling that keeps deep networks trainable Lesson

Layer normalization rescales the numbers flowing through each layer of a network to a consistent range, which stops signals from exploding or vanishing as they pass through dozens of layers -- and it is what makes training a transformer at all possible.