Learn · Beginner

Regularization: why deliberately handicapping a model makes it better

Regularization is any technique that makes a model do worse on its training data so that it does better on data it has never seen. That trade sounds like a mistake and is in fact the central discipline of machine learning. The two techniques that matter most -- dropout and weight decay -- both work by handicapping the network during training, and together they made deep learning practical.

The problem they solve has a name: overfitting. A model that overfits has memorized its training examples rather than learning the pattern behind them.

Why memorization is the default

Here is the fact that makes the whole field uncomfortable, established by Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht and Oriol Vinyals in 2016: a standard neural network can memorize a dataset with completely random labels. Shuffle the labels on a million images so that "cat" and "airplane" are assigned by coin flip -- destroying every pattern -- and the network will still drive its training error to zero. It has enough capacity to just remember all million of them.

So a network achieving perfect training accuracy tells you nothing. It might have found the concept of a cat. It might have built an elaborate lookup table. From the training loss, these look identical. They differ only when you show it a cat it has never seen.

The analogy is a student who has the last decade's exam papers with the answer key. They can score 100 percent on all of them. Whether they understand the subject or memorized ten answer keys is a question the practice exams cannot answer, and the real exam answers immediately. Regularization is the teacher taking the answer key away.

Dropout: randomly break the network

Dropout, introduced by Geoffrey Hinton and colleagues in 2012 and developed by Nitish Srivastava and coauthors in 2014, is one of the strangest good ideas in the field. On every training step, randomly switch off some fraction of the neurons -- typically half. Not permanently. A different random half each time. At test time, use the whole network.

Why does sabotaging your own model help? Because it kills co-adaptation. In a network trained normally, neurons form fragile private arrangements: neuron 400 learns to detect something only if neuron 712 handles a specific correction, and neither works alone. These conspiracies are brittle and they are how memorization gets implemented.

Dropout makes them impossible. No neuron can rely on any other neuron being present, because any collaborator may vanish on the next step. The only strategy that survives is for each neuron to learn something independently useful, and for the knowledge to be spread redundantly across many of them.

Srivastava's framing is that dropout trains an ensemble. A network with n neurons contains an exponential number of possible sub-networks, and dropout samples a different one each step, training all of them with shared weights. At test time, using the full network approximately averages that ensemble -- and averaging many models is the oldest reliable way to reduce error in statistics.

The organizational version: a team where every person is randomly out sick half the time cannot develop a single point of failure. It is less efficient day to day and far more robust.

Weight decay: make every parameter pay rent

Weight decay is quieter and has outlasted dropout. On every update, shrink every parameter slightly toward zero. Any parameter that is not being actively pushed away from zero by the data will drift back to it.

This makes size a cost rather than a default. A parameter can only stay large if the training signal keeps re-earning its magnitude. Weights encoding a genuine, repeated pattern get reinforced constantly and survive. Weights encoding one training example's quirk get reinforced once and then quietly decay away. It is a rent system: pay every month or lose the apartment.

Mathematically this expresses a preference for simpler models -- smaller weights mean a smoother function, and a smooth function cannot bend to hit every individual training point. It is Occam's razor implemented as a subtraction in the optimizer loop.

There is an important subtlety that took the field years to notice. In adaptive optimizers like Adam, weight decay was implemented by folding it into the gradient, which meant Adam's per-parameter scaling silently distorted it. Ilya Loshchilov and Frank Hutter showed in 2017 that decoupling the decay from the gradient update fixes this. The result is AdamW, and it is why essentially every large model you have heard of -- including every model in today's news -- was trained with a W on the end of its optimizer.

What changed for large models

Modern language models mostly dropped dropout. The reason is data. Dropout was vital when a model had far more capacity than its dataset had examples -- when memorization was the path of least resistance. A frontier model trains on trillions of tokens and often sees a given passage once. You cannot memorize what you see once, so there is nothing for dropout to prevent. Weight decay stayed, because the preference for simplicity is useful regardless.

The theory also got weirder. Classical statistics predicted that a model much larger than its dataset would overfit catastrophically. Preetum Nakkiran and colleagues documented deep double descent: performance gets worse as models grow, then, past a certain point, gets better again -- indefinitely. That result is a large part of why the field kept scaling instead of shrinking, and why grokking, where a model memorizes for thousands of steps and then abruptly generalizes, is possible at all. Regularization is still essential and still not fully understood.

Key papers
Dropout: A Simple Way to Prevent Neural Networks from Overfitting (Srivastava et al., 2014)
Improving Neural Networks by Preventing Co-adaptation of Feature Detectors (Hinton et al., 2012)
Decoupled Weight Decay Regularization (Loshchilov & Hutter, 2017)
Understanding Deep Learning Requires Rethinking Generalization (Zhang et al., 2016)
Deep Double Descent (Nakkiran et al., 2019)

Key questions

What problem does regularization solve?

It solves overfitting, where a model memorizes the specific examples it was trained on instead of learning the general pattern behind them, and then fails on anything new. Regularization deliberately makes memorization harder so the model is forced toward rules that actually generalize.

How is dropout different from weight decay?

Dropout randomly switches off a fraction of the network's neurons during each training step, forcing the model to build redundant, non-fragile representations. Weight decay instead continuously shrinks every parameter toward zero, so a parameter only stays large if it repeatedly earns its size.

Do large language models still use dropout?

Mostly no. Dropout was essential when models were large relative to their datasets, but frontier language models train on trillions of tokens and rarely see the same text twice, so there is little to memorize. Weight decay, by contrast, is still standard in essentially every large training run.

Cite this

APA

Ground Truth. (2026, July 15). Regularization: why deliberately handicapping a model makes it better. Ground Truth. https://groundtruth.day/learn/regularization-dropout-and-weight-decay.html

BibTeX

@misc{groundtruth:regularization-dropout-and-weight-decay,
  title  = {Regularization: why deliberately handicapping a model makes it better},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {jul},
  url    = {https://groundtruth.day/learn/regularization-dropout-and-weight-decay.html}
}

Topics: fundamentals · training · overfitting · dropout · weight-decay