Ground Truth.
AI, checked against the source.

Learn · Intermediate

Regression to the mean: why extreme AI results often move back toward normal

Regression to the mean is the tendency for an unusually high or low measurement to be followed by one closer to its typical level when chance contributed to the extreme result. It matters in AI evaluation because a model or intervention can appear to improve simply because it was measured immediately after an unusually bad run.

Imagine a student whose ordinary quiz performance varies from day to day. On the worst quiz of the term, difficult questions and bad luck may combine with their actual preparation. If the student studies in a new way and scores closer to normal on the next quiz, the improvement is real as a change in scores, but the whole improvement cannot automatically be credited to the new study method. Part of the low first score was an unusually unfavorable draw.

The same mechanism appears when researchers choose a model because it failed spectacularly on a small benchmark, apply a fix, and retest. It also works in the other direction: a system selected because it won an unusually noisy competition may score lower on a fresh test even if its underlying ability has not changed. There is no mysterious force pushing performance toward mediocrity. Selection picked an observation containing an unusually large temporary component; that component is less likely to recur on an independent measurement.

Signal, noise and selection

A measured result combines stable influences with temporary variation. The stable part might include the model, training, evaluation protocol and task distribution. Temporary variation might include which examples were sampled, random generation choices, evaluator judgments or infrastructure hiccups. The distinction is conceptual: some sources of variation persist across a particular repeated test and others do not. If the evaluation changes, even the definition of the underlying average can change.

Suppose a model’s long-run success rate on a fixed distribution is 70%. In a small batch it might score 55% by chance. Selecting that batch because it is unusually low makes a subsequent batch likely to look better, even without an update. The example is illustrative, not a result from a study. The effect becomes stronger when measurements are noisy or the selected result is especially extreme; it becomes smaller when the original observation is a precise measure of a persistent difference.

Regression to the mean is related to, but distinct from, selection bias. Selection bias describes a broader family of distortions produced by how observations enter a sample. Regression to the mean describes a particular pattern when selection on an extreme noisy observation is followed by another measurement. It is also different from a genuine loss of skill, a change in task difficulty, or a system drifting over time. Each can produce a similar-looking graph.

How a control group helps

To test whether a change actually helps, compare a treated system with a suitable control measured over the same period. Random assignment can balance sources of variation in expectation. If both groups were selected after a poor measurement and both recover similarly, the common recovery is a warning against crediting the intervention. If the treated group improves more, the difference can be evidence of an effect, subject to sampling uncertainty and the quality of randomization.

Donald Rubin’s 1974 paper on causal effects provides a framework for asking what would have happened under a different treatment. The lesson on randomized trials explains why the comparison matters. A control is not a magic eraser: a tiny sample can still give an unstable estimate, treatment and control can affect one another, and a result may not transfer to a different population. Report the selection rule, the full distribution of scores, the number of cases and uncertainty, rather than only the most dramatic before-and-after example.

A fresh holdout is useful when selecting among many models or prompts. The winning configuration may owe some of its observed lead to luck. Evaluate the chosen configuration on data and runs that did not determine the winner; where possible, compare methods on the same examples using paired comparisons. This reduces noise from differences in example difficulty, although it does not remove every source of bias. Repeated testing against the same holdout eventually makes it part of the selection process.

Why this matters for AI news

The METR assessment of AI research acceleration illustrates why trends need more than two striking points: METR fits a benchmark trend and performs robustness checks while acknowledging uncertainty and limits on public access to underlying evidence. That assessment is not itself presented here as a regression-to-the-mean error. It shows why readers should ask how observations were selected, how many there were, and whether a reported change survives alternative reasonable analyses.

Regression to the mean does not mean every improvement is fake. A new training method can work, and an unusually poor result can reveal a real defect. The lesson is narrower and more useful: before assigning a cause to a recovery from an extreme, establish what would likely have happened without the intervention. A controlled comparison and fresh measurements are usually more persuasive than a dramatic second score.

Key papers
Donald Rubin, Estimating causal effects of treatments in randomized and nonrandomized studies (1974)

Key questions

Does regression to the mean mean a model necessarily gets worse?

No. A later measurement is often less extreme because the original result included unusually favorable or unfavorable variation.

How can researchers distinguish a real improvement from regression to the mean?

Compare the intervention with an appropriate control over the same period, and test on fresh data or runs.
Cite this

APA

Ground Truth. (2026, November 17). Regression to the mean: why extreme AI results often move back toward normal. Ground Truth. https://groundtruth.day/learn/regression-to-the-mean.html

BibTeX

@misc{groundtruth:regression-to-the-mean,
  title  = {Regression to the mean: why extreme AI results often move back toward normal},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {nov},
  url    = {https://groundtruth.day/learn/regression-to-the-mean.html}
}

Topics: evaluation · statistics · experiments