Learn · Beginner

Ensembles: why averaging several models beats picking the best one

An ensemble is a group of models that all answer the same question, with their answers combined -- usually by averaging or voting -- into a single prediction. It is the most reliable free lunch in machine learning: an ensemble almost always beats its own best member, it needs no new ideas or data, and it is how probabilistic forecasts get their probabilities. When Google DeepMind's weather model says there is an 80% chance a hurricane arrives as a Category 5, that number is the fraction of an ensemble's members that predicted one.

The core intuition

Ask one person to guess how many sweets are in a jar and they will be off. Ask two hundred and average the guesses, and the average lands remarkably close -- a result Francis Galton documented at a country fair in 1907. Individual errors are partly independent: some guess high, some low, and the over- and under-shoots cancel while the real information everyone is using survives.

Machine learning models behave the same way, with one condition: the members have to make different mistakes. Two identical models averaged together give you exactly the original model. The whole benefit lives in the disagreement.

The arithmetic explains everything else. Average N models whose errors are independent and equally sized, and the ensemble's error variance is 1/N of a single model's -- four models, a quarter of the variance. In practice errors are correlated, since members train on similar data and share architectural biases, so you never get the full 1/N. But you always get some of it, and how much is a direct measure of how genuinely different the members are.

How you get models that disagree

Bagging. Leo Breiman's 1996 paper introduced bootstrap aggregating: train each member on a random resample of the training data, drawn with replacement. Each model sees a slightly different world and learns slightly different quirks. Random forests, which Breiman published five years later, add a second source of disagreement -- each tree also considers only a random subset of features at each split -- and were the strongest general-purpose method on tabular data for over a decade.

Boosting. Instead of training members independently, train them in sequence, each one focused on the examples its predecessors got wrong. This produces a very different kind of ensemble, one where members specialise rather than merely vary. Gradient-boosted trees are still the default choice for structured, tabular problems in 2026.

Deep ensembles. For neural networks, the astonishing finding is that you barely need to try. Train the same architecture on the same data several times, changing nothing but the random seed that sets the initial weights, and you get members that disagree usefully. Lakshminarayanan and colleagues showed in 2016 that this simple recipe gives accuracy gains and, more importantly, well-behaved uncertainty estimates. Later work explained why: different random starts land in genuinely different regions of the loss landscape, so the resulting networks are not small perturbations of each other -- they are different solutions that happen to fit the same data.

The part that matters most: uncertainty

Accuracy is the advertised benefit. Calibration is the more valuable one.

A single neural network is famously overconfident. It will assign 95% probability to a wrong answer without hesitation, because nothing in ordinary training teaches a model what it does not know. An ensemble gets this almost for free. When all members agree, the input resembles the training data and confidence is warranted. When they scatter, you are somewhere the training data did not cover -- and the spread is the uncertainty estimate.

This is why ensembles dominate forecasting. Weather centres have run ensemble forecasts since the early 1990s: perturb the starting conditions slightly, run the simulation many times, and report the distribution rather than a single trajectory. Google DeepMind's WeatherNext does the same thing with a learned model instead of a physics simulation, generating hundreds of possible weather scenarios from one starting point. Its technical write-up, Skillful joint probabilistic weather forecasting from marginals, describes training on individual variables and then recovering the relationships between them through injected noise plus ensembling. The probability attached to a hurricane's intensity is not something the model outputs. It is something counted across the ensemble.

The cost, and the ways around it

Running ten models costs ten times as much as running one. For a weather forecast on a datacentre, fine. For a chatbot answering millions of requests, not fine -- which is why frontier language models are not deployed as ensembles.

Three workarounds are common. Distillation, from Hinton, Vinyals and Dean, trains one small model to imitate the ensemble's averaged outputs, capturing much of the benefit at single-model cost. Model merging averages the weights rather than the outputs -- cheaper, but a more fragile operation that only works when the models are close relatives. Mixture-of-experts models look like ensembles but are not: a router picks a few experts per token instead of running all of them and averaging, which buys capacity rather than error cancellation.

A lot of modern inference is also quietly ensemble-shaped. Sampling several answers from one model and taking the majority -- self-consistency -- is an ensemble over random decoding paths rather than over models, and much of what is called test-time compute is the same trick: generate variation, then aggregate.

When it does not help

Ensembling cannot fix a bias every member shares. If all your models trained on the same skewed dataset, they will all be wrong in the same direction, and averaging preserves that error perfectly -- it only cancels the parts that differ. This is the failure mode to watch: an ensemble that agrees confidently and is confidently wrong looks exactly like an ensemble that agrees confidently and is right. Diversity is not a nice-to-have you can skip. It is the entire mechanism.

Ground Truth. (2026, August 8). Ensembles: why averaging several models beats picking the best one. Ground Truth. https://groundtruth.day/learn/ensembles-and-why-averaging-predictions-works.html

@misc{groundtruth:ensembles-and-why-averaging-predictions-works,
  title  = {Ensembles: why averaging several models beats picking the best one},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/learn/ensembles-and-why-averaging-predictions-works.html}
}

Topics: ensembles · fundamentals · uncertainty · forecasting · machine-learning