Learn · Intermediate

Residual Connections: The Shortcut That Made Deep Networks Possible

A residual connection is a shortcut that adds a layer's input directly to its output. Instead of forcing each layer to reproduce the entire signal from scratch, you let it learn only the change — the residual — and add that change to what came in. This one-line idea, introduced by Kaiming He and colleagues at Microsoft Research in 2015, is the reason neural networks can be hundreds of layers deep without falling apart, and it is a load-bearing part of every modern transformer.

The problem it solved. In the early 2010s, everyone believed deeper networks should be more powerful — more layers, more capacity. But in practice, past a couple dozen layers, accuracy got worse, and not because of overfitting: even the training error rose. This was the "degradation problem." Something about depth itself was breaking training. The culprit is tied to how networks learn: gradient descent pushes error signals backward through every layer via backpropagation, and across dozens of layers those signals tend to shrink toward zero (vanishing gradients) or blow up (exploding gradients). By the time the signal reaches the early layers, it's too faint or too chaotic to teach them anything.

The fix. He and colleagues reframed what a layer should do. Rather than learn a target mapping H(x) directly, a block learns a residual function F(x), and the network computes F(x) + x — the block's output plus a clean copy of its input, carried straight across by the shortcut. If the best thing a layer can do is nothing, it just learns F(x) = 0, and the identity passes through untouched. That turns out to be far easier to learn than reconstructing the identity from scratch, which is exactly what deep plain networks struggle with. The famous result, ResNet, trained networks 152 layers deep — an order of magnitude deeper than what worked before — and won the 2015 ImageNet competition.

The intuition. Picture editing a long document by passing it down a line of editors. In a plain network, each editor must rewrite the whole document from memory and hand on their version — errors compound and the original meaning erodes. With residual connections, each editor receives the actual document and writes only their edits in the margin; the original always passes through intact, and any editor who has nothing to add simply changes nothing. The text can survive a hundred editors because it is never reconstructed, only refined.

Why gradients love it. The shortcut also gives the backward pass a clean highway. Because the input is added straight to the output, the gradient can flow backward through the addition without being repeatedly multiplied down by each layer's weights. Early layers keep receiving a strong, usable learning signal no matter how deep the network gets. This is why the closely related Highway Networks, proposed the same year, used learned gates to control the shortcut — ResNet's insight was that a plain, ungated identity shortcut works even better and simpler.

Residuals in transformers. When the transformer arrived in 2017, it inherited this idea wholesale. In every transformer block, the attention sublayer and the feed-forward sublayer are each wrapped as output = sublayer(x) + x, usually paired with normalization. This is not a minor detail — it is a large part of why you can stack a transformer 32, 80, or more layers deep and still train it. Every large language model you have used is, structurally, a tall tower of residual blocks. The residual stream — the running sum that each block reads from and writes back into — has even become a central object in mechanistic interpretability, where researchers trace how information accumulates layer by layer.

Why it matters. Residual connections are one of the quiet enablers of the entire deep-learning era. Depth is where much of a network's power comes from, and before residuals, depth was a trap. After them, depth became a dial you could turn — the precondition for the scaling laws that drive today's ever-larger models. It is a rare example of a change so simple you can write it in a few characters, and so consequential that nearly every serious architecture since has kept it.

The honest caveat. Residual connections make deep networks trainable, not automatically good — they don't add capacity so much as let you use the capacity depth provides. They interact subtly with normalization and initialization, and getting that interaction wrong can still destabilize training. And while they tame vanishing gradients, very deep or very wide models raise fresh challenges around compute, memory, and diminishing returns. The shortcut opened the door to depth; walking through it well is still engineering.

Ground Truth. (2026, July 8). Residual Connections: The Shortcut That Made Deep Networks Possible. Ground Truth. https://groundtruth.day/learn/residual-connections.html

@misc{groundtruth:residual-connections,
  title  = {Residual Connections: The Shortcut That Made Deep Networks Possible},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/residual-connections.html}
}

Topics: residual-connections · deep-learning · transformers · training · architecture