normalization
Everything on Ground Truth tagged “normalization” — 2 items.
Batch normalization: grading every layer on a curve so deep networks train faster Lesson
Batch normalization standardizes each layer's intermediate values using the current training batch's average and spread, then lets the network learn its own scale and offset; introduced in 2015, it let an image classifier reach the same accuracy in 14 times fewer training steps.
Layer normalization: the rescaling that keeps deep networks trainable Lesson
Layer normalization rescales the numbers flowing through each layer of a network to a consistent range, which stops signals from exploding or vanishing as they pass through dozens of layers -- and it is what makes training a transformer at all possible.