relu

Activation functions: the tiny nonlinearity that makes deep learning possible Lesson

An activation function is a simple nonlinear operation applied to each number flowing through a neural network, and without one, stacking a hundred layers would be mathematically identical to using a single layer. Modern transformers mostly use gated variants like SwiGLU, which Noam Shazeer showed outperform plain ReLU.