Learn · Intermediate

Sequential testing: how to watch a changing model without chasing noise

Sequential testing is a way to evaluate evidence as observations arrive while controlling the chance of a false alarm across repeated checks. It matters because repeatedly checking an ordinary statistical test and stopping at the first apparently significant result can make random noise look like a real change.

Imagine checking whether a coin is fair after every toss. Even a fair coin will occasionally produce a run of heads. If you look often enough and announce a discovery at the first unusual run, your chance of a false alarm is greater than the one-check error rate printed on a conventional test. Sequential methods plan for the entire checking process, not merely the last look.

Why repeated looks change the question

A conventional hypothesis test asks how surprising a result would be under a stated baseline, with a particular analysis plan. A 5% false-positive rate is a statement about repeated experiments following that plan; it is not a statement that there is a 5% chance the hypothesis is true. If a team runs many unadjusted tests as data accumulates, “at least one test crosses the threshold” is a different event from “the one scheduled test crosses it.” The risk depends on dependence between checks and the stopping rule, not simply the number of times somebody glanced at a chart.

Abraham Wald’s Sequential Tests of Statistical Hypotheses established a foundational framework for making decisions as evidence accumulates. In a sequential probability ratio test, one compares how likely the observations are under two specified hypotheses. Evidence crossing an upper boundary favors one, crossing a lower boundary favors the other, and remaining between boundaries means collecting more data. The boundaries are chosen for desired error rates under the method’s assumptions. It is a procedure for a sequence of decisions, not a license to choose a threshold after seeing the result.

A concrete AI example

Suppose a team suspects that a hosted model has changed. It freezes 100 representative tasks and evaluates each day against a preselected baseline. A single bad day might reflect sampling variation, changes in task mix, or an actual regression. If the team publishes an alarm whenever any daily result passes an ordinary 5% test, the alarm rule can overstate the evidence after many days.

One response is to specify evaluation windows, a minimum effect worth detecting, a control series and a rule for when to announce a change. The independent LiveNerf design illustrates preregistration: its reported rule requires two consecutive windows with a 99% interval excluding zero, a sufficiently large effect and a control check. That specific rule should be interpreted according to its own design and validation, not assumed to provide a universal false-alarm guarantee. Its fixed academic questions also do not cover every coding-agent task. The paired comparisons lesson explains how repeatedly using the same cases can reduce noise without fixing a nonrepresentative panel.

Boundaries, budgets and always-valid evidence

Different designs solve different versions of the problem. A group-sequential design schedules a small number of interim analyses and chooses more demanding early thresholds, preserving a specified overall error rate when its assumptions hold. An alpha-spending design allocates the allowable false-positive budget over planned looks. An always-valid approach constructs evidence or confidence sequences intended to remain valid under permitted stopping times. These are related ideas, not interchangeable recipes; assumptions about independence, sampling, dependence and the baseline matter.

For example, an evaluation could reserve most of its false-alarm budget for a final analysis and a smaller portion for an early check. The early boundary is then harder to cross. A severe early result may justify action, while a borderline result leads to continued observation. Designing the boundaries before looking at outcomes protects the interpretation of the final claim.

There is a tradeoff. A more conservative alarm can be slower to detect a real change, particularly a small one. Statistical power depends on sample size, variability, the effect size and the decision rule. “No statistically significant change” therefore does not mean “no change”; it may mean the study cannot distinguish a smaller change from noise. The calibration and confidence lesson provides related context for interpreting uncertainty.

What sequential testing cannot repair

A perfect stopping rule cannot repair biased measurement. If the questions are unrepresentative, answer keys are wrong, or the interface changes midstream, an apparently rigorous signal may describe the harness rather than the model. Neither does a statistically detectable difference automatically matter to users: define the smallest practically meaningful change in advance. Conversely, harm in a rare but important workflow may deserve a targeted investigation even if an average score stays stable.

The practical sequence is to define the decision, choose representative tasks and a baseline, set an error-controlled monitoring rule, record changes to the environment, and report both uncertainty and the limits of detection. Sequential testing turns repeated observation into a designed process; it cannot turn a narrow measurement into a universal verdict.

Ground Truth. (2026, November 9). Sequential testing: how to watch a changing model without chasing noise. Ground Truth. https://groundtruth.day/learn/sequential-testing-and-early-stopping.html

@misc{groundtruth:sequential-testing-and-early-stopping,
  title  = {Sequential testing: how to watch a changing model without chasing noise},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {nov},
  url    = {https://groundtruth.day/learn/sequential-testing-and-early-stopping.html}
}

Topics: evaluation · statistics · experimentation