Learn · Intermediate

Equivalence testing: how to show two AI systems are close enough

Equivalence testing is a statistical method for asking whether two systems are close enough for a practical purpose. It matters in AI because “we did not find a significant difference” does not mean a cheaper model, a human and an AI tutor, or two safety methods truly perform about the same. An equivalence test requires researchers to define what difference would matter before looking at the result, then show that the plausible gap stays inside that boundary.

Imagine comparing a new AI tutor with a human tutor. A conventional difference test begins with a default stance: there is no difference, and it looks for enough evidence to reject that stance. If the experiment is small or results are noisy, it may fail to reject it. That failure does not tell us whether the tutors are nearly identical, slightly different, or dramatically different in a way the study could not see. “Absence of evidence is not evidence of absence” is not a slogan here; it is a property of the test design.

Equivalence reverses the useful question. Before collecting outcomes, choose a smallest effect size of interest: a gap so large that a decision-maker would care. For an exam scored out of 27, perhaps a two-percentage-point drop is acceptable but a five-point drop would change a purchasing decision. The range between minus two and plus two is the equivalence margin. Researchers then estimate the difference and its uncertainty. If the entire uncertainty interval falls inside that margin, they can say the data support practical equivalence at the chosen standard.

The usual procedure is called two one-sided tests, or TOST. Rather than one test asking “is the difference exactly zero?”, it asks two questions: is the new option meaningfully worse than the lower boundary, and is it meaningfully better than the upper boundary? Passing both rejects differences outside the acceptable interval. Robert Schuirmann developed the approach, and Daniël Lakens’s practical primer explains why it became important for applied work.

Confidence intervals provide the clearest visual. Suppose an AI-minus-human score difference is -0.6 percentage points, with a 90% confidence interval from -2.2 to 1.0. If the predefined equivalence margin is plus or minus three points, the entire interval fits inside: practical equivalence is supported. If the margin is plus or minus one point, it does not: the same observed average is too uncertain to make the stronger claim. The result depends on a decision rule chosen for a reason, not on whether the average happened to be close to zero.

This is why the StudentBench result is more informative than a headline saying AI and humans “tied.” Its pooled adjusted AI-minus-human estimate was -0.58 points, with a 90% interval from -2.18 to 1.03, and the authors assessed it against stated equivalence bounds. That supports an immediate-GRE-gain claim under their protocol. It does not establish equivalence for child motivation, long-term retention, verbal learning under every analysis, or the whole social role of a tutor. Equivalence always has a target outcome, a margin and a population.

The hardest and most important step is selecting the margin. It should come from stakes, prior evidence, costs, safety requirements or user benefit—not from picking a wide interval after seeing unfavorable data. A hospital might require a very narrow margin for a diagnostic tool. A company routing easy customer questions to a cheaper model might accept a wider margin if users can escalate. In AI evaluation, margins should also acknowledge failure asymmetry: a tiny average gap can conceal rare but unacceptable errors.

Equivalence is not the same as non-inferiority. Non-inferiority asks only whether a new option is not worse by more than a margin; it does not care whether it might be substantially better. Equivalence needs both sides. Nor is it the same as a benchmark tie. A leaderboard may show two rounded scores that look identical while uncertainty, task mixture and failure modes differ sharply. Ground Truth’s lessons on calibration, holdout sets, and null baselines explain adjacent pieces of this discipline.

Use equivalence testing when the decision is genuinely about substitution: can a small model replace a large one for this task, can a generated dataset stand in for a real one within defined bounds, or can an AI tutor provide enough immediate practice at a specified cost? State the margin, report the interval, and name what was not tested. That converts “no meaningful difference” from a vague press-release phrase into a claim a reader can inspect.

Ground Truth. (2026, September 25). Equivalence testing: how to show two AI systems are close enough. Ground Truth. https://groundtruth.day/learn/equivalence-testing-when-no-difference-is-not-enough.html

@misc{groundtruth:equivalence-testing-when-no-difference-is-not-enough,
  title  = {Equivalence testing: how to show two AI systems are close enough},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {sep},
  url    = {https://groundtruth.day/learn/equivalence-testing-when-no-difference-is-not-enough.html}
}

Topics: evaluation · statistics · benchmarks · education