Learn · Beginner

Activation functions: the tiny nonlinearity that makes deep learning possible

An activation function is a small nonlinear operation applied to every number as it passes through a neural network layer. It is the reason depth works at all: without one, stacking a hundred layers is mathematically identical to using a single layer, because multiplying matrices together just produces another matrix. Modern transformers overwhelmingly use gated variants like SwiGLU, following Noam Shazeer's 2020 finding that they consistently beat the simpler ReLU.

That first sentence hides the most important idea in the topic, so it is worth making concrete. A neural network layer, stripped down, multiplies its input by a weight matrix. If you stack two such layers, you compute W2 x (W1 x input). But matrix multiplication is associative, so that equals (W2 x W1) x input - and W2 x W1 is just another matrix. Your two-layer network is a one-layer network wearing a costume. Add ninety-eight more layers and nothing changes. The entire network can only draw straight lines, no matter how big it gets.

Put a nonlinear function between the layers and the collapse becomes impossible. Now the composition genuinely builds on itself: the second layer operates on something the first layer bent, and the space of functions the network can represent explodes. This is the actual content of the universal approximation theorems - given a nonlinearity, a sufficiently wide network can approximate essentially any continuous function.

The workhorse: ReLU

The rectified linear unit is almost comically simple. If the number is positive, keep it. If it is negative, output zero. That is the whole function.

It looks too crude to matter, and for years the field used smooth S-shaped curves instead - the sigmoid and the hyperbolic tangent - on the intuition that biological neurons and good mathematics both prefer smoothness. Xavier Glorot, Antoine Bordes and Yoshua Bengio helped establish in 2011 that the crude version wins, and the reason is about training rather than expressiveness.

Neural networks learn through backpropagation, which sends an error signal backward through every layer, multiplying by each function's slope along the way. Sigmoid and tanh flatten out at both extremes - their slope approaches zero for large positive or negative inputs. Multiply a dozen near-zero slopes together and the error signal arriving at the early layers is effectively nothing. Those layers stop learning. This is the vanishing gradient problem, and it is why deep networks were so hard to train before the 2010s.

ReLU's slope for positive inputs is exactly 1. The error signal passes through untouched, however many layers deep. It is also nearly free to compute - a comparison against zero - and it produces sparse activations, since roughly half its outputs are zero. Its one real flaw is the dying ReLU: a unit pushed permanently negative outputs zero forever, gets zero gradient, and never recovers. Leaky ReLU, which gives negative inputs a small nonzero slope instead of flat zero, exists mostly to patch this.

The smooth successor: GELU

Dan Hendrycks and Kevin Gimpel proposed the Gaussian Error Linear Unit in 2016, and it became the default in early transformers including BERT and the GPT series. GELU behaves like a softened ReLU: instead of an abrupt cutoff at zero, it curves gently, letting slightly negative values through in diminished form.

The motivation is probabilistic. GELU weights each input by the probability that a standard normal random variable falls below it - so the function asks "how likely is this value to matter?" rather than applying a hard threshold. In practice the practical benefit is the smoothness: a continuous derivative everywhere gives optimisers like Adam a better-behaved surface to descend, which matters more as models get deeper. It also sidesteps the dying-unit problem, since the gradient never becomes exactly zero.

A related function, Swish (also called SiLU), emerged from an automated search by Prajit Ramachandran, Barret Zoph and Quoc Le at Google Brain. They let a search algorithm explore the space of possible activation functions and it rediscovered something almost identical to GELU - a satisfying independent confirmation that the smooth-ReLU shape is genuinely a good region of the design space.

The current default: SwiGLU

The activation in most large models today is not a single curve at all. SwiGLU is a gated unit, and the idea is a structural change rather than a new shape.

Instead of one path through the feedforward layer, SwiGLU creates two. One path passes through a Swish curve; the other passes through unchanged. The two are then multiplied element by element. The Swish path acts as a learned gate: where it outputs near zero, it shuts the other path off; where it outputs large values, it lets information through amplified.

The analogy is a dimmer switch that the network learns to operate per-value. ReLU is a plain on-off switch with a fixed threshold. SwiGLU learns, for each position, how far to open the valve - and crucially, that decision depends on the input rather than being baked into the function's shape.

Shazeer's 2020 paper tested several gated variants against the standard options on transformer pretraining and found the gated ones consistently better. The paper's closing line is famously wry - he attributes the success to "divine benevolence", which is a researcher's honest way of saying the empirical result is solid and the theoretical explanation is not. That candour is worth noting: this is a case where the field adopted something because it measurably works, not because anyone can fully explain why.

The practical cost is that gating needs three weight matrices in the feedforward block instead of two. Implementations compensate by shrinking the hidden dimension, keeping the parameter count roughly matched. You will find SwiGLU inside the transformer feedforward blocks of Llama, Qwen, Gemma and most other current open models - and in the nonlinear building blocks of tools like torchwright, which constructs transformer weights directly rather than training them.

One last thing worth keeping straight: the activation function inside the network is a different creature from the softmax at the output. Activations shape how information flows through hidden layers; softmax turns final scores into a probability distribution over tokens. They are both nonlinearities, but they answer different questions.

Ground Truth. (2026, July 24). Activation functions: the tiny nonlinearity that makes deep learning possible. Ground Truth. https://groundtruth.day/learn/activation-functions.html

@misc{groundtruth:activation-functions,
  title  = {Activation functions: the tiny nonlinearity that makes deep learning possible},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/activation-functions.html}
}

Topics: fundamentals · neural-networks · transformers · relu · gelu · swiglu · architecture