Learn · Beginner

The bias-variance tradeoff: why a model can fail by being too simple or too clever

Every prediction error a model makes can be split into two competing sources: bias, the error from being too rigid to capture the real pattern, and variance, the error from being so flexible that it fits the random noise in your particular training sample. Reducing one usually increases the other. Choosing where to sit on that tradeoff is not a preliminary step before the real machine learning -- it is most of the real machine learning.

The name comes from a decomposition made precise by Stuart Geman, Elie Bienenstock and Rene Doursat in their 1992 paper on the bias/variance dilemma in neural networks, and it is the organizing idea of Hastie, Tibshirani and Friedman's The Elements of Statistical Learning, the standard graduate text.

The picture that explains it

Imagine you are trying to draw the relationship between a person's height and their weight, from thirty measured people.

Draw a horizontal line at the average weight. That is maximum bias: your model is so simple it ignores height entirely. It is wrong in a consistent, systematic way. But it is also completely stable -- collect thirty different people and you will get almost the same line.

Now draw a wiggly curve that passes exactly through all thirty points. Zero training error. That is maximum variance: your curve has bent itself around every measurement error, every unusually heavy person, every rounding artifact. Collect thirty different people and you get a wildly different curve. The model has learned your sample rather than the world.

The straight-line fit sits in between, and that is not a compromise -- for this problem it is genuinely the best answer, because it captures the real relationship without pretending the noise is signal.

Why they trade off

The two error sources move in opposite directions as you change model flexibility, and there is a reason for that beyond coincidence.

Flexibility is the capacity to produce many different functions. A model that can only produce straight lines has almost no capacity to chase noise -- but also no capacity to represent a curve. A model that can produce any function at all can represent anything, including the exact pattern of measurement errors in your sample. You cannot have the second kind of freedom without also having the first kind of danger. Every increase in what the model can express is also an increase in what it can mistakenly express.

So total error typically traces a U. Start too simple and bias dominates. Add flexibility and total error falls. Keep going and variance takes over and error climbs again. The bottom of that U is what you want, and you cannot see it from training error alone -- training error just falls forever. You need held-out data, which is exactly why every serious evaluation splits the data before touching it. See how AI is benchmarked for why that split gets contaminated so easily.

What this explains in practice

Most of the standard toolkit is a bias-variance lever in disguise.

Regularization -- weight decay, dropout, early stopping -- deliberately handicaps the model, buying a little bias to cut a lot of variance. Data augmentation attacks variance from the other side, by making the training sample look more like the world. Ensembles work because averaging many high-variance models cancels their independent errors while leaving the shared signal intact -- that is a variance reduction, and it is why random forests exist. More training data reduces variance without touching bias, which is why "get more data" is such a reliable answer.

It also explains a failure that confuses people constantly: a model that scores beautifully in development and badly in production. That is high variance meeting a slightly different world. And its mirror image -- a model that is mediocre everywhere, including on training data -- is high bias, which more data will not fix.

Where it gets strange

The classic U-curve was formulated for models smaller than their training sets. Modern deep networks routinely have far more parameters than training examples, and by the classical story they should be catastrophically high-variance. They are not.

Mikhail Belkin and colleagues documented what actually happens in Reconciling modern machine-learning practice and the classical bias-variance trade-off, and Preetum Nakkiran and colleagues at OpenAI extended it in Deep Double Descent. Test error follows the expected U up to the point where the model can exactly fit the training data -- and then, as you keep making it bigger, error falls again, sometimes below the classical minimum. The curve has a second descent.

The working explanation is that among the enormous number of ways an overparameterized model could fit the data, gradient descent tends to find unusually smooth solutions, so extra capacity buys better solutions rather than more noise-fitting. This does not repeal the tradeoff -- bias and variance are still what error decomposes into -- but it does mean "bigger model, more overfitting" is not a safe rule of thumb anymore. Nakkiran's paper also shows the effect appears along the training-time and dataset-size axes, not just model size, which is one reason scaling laws behave the way they do.

The takeaway

When a model underperforms, ask which error you have. If it is bad on the training data too, you have a bias problem: it needs more capacity, better features, or a different architecture. If it is excellent on training data and poor on held-out data, you have a variance problem: it needs more data, more regularization, or less capacity. Those two diagnoses lead to opposite actions, and getting them backwards is the single most expensive mistake in applied machine learning. Related reading: grokking, ablation studies, and shortcut learning.

Ground Truth. (2026, August 24). The bias-variance tradeoff: why a model can fail by being too simple or too clever. Ground Truth. https://groundtruth.day/learn/bias-variance-tradeoff.html

@misc{groundtruth:bias-variance-tradeoff,
  title  = {The bias-variance tradeoff: why a model can fail by being too simple or too clever},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/bias-variance-tradeoff.html}
}

Topics: fundamentals · statistics · generalization · overfitting · model-selection