Learn · Intermediate

Paired comparisons: how to tell whether one AI system really beats another

A paired comparison evaluates two AI systems on the same examples and studies how their results differ on each example. It can reveal whether one system consistently improves on another and usually gives a more informative comparison than treating the two sets of scores as unrelated.

Imagine two coding assistants asked to fix the same 100 bugs. Assistant A passes 70, and assistant B passes 75. The five-point gap does not tell you whether B fixed five additional bugs while matching A on all others, or whether each assistant solved a largely different set. Both outcomes produce the same overall scores, but they imply different choices for users and different amounts of uncertainty about the gap.

The key is to preserve the identity of each task. For every bug, record whether A succeeded and whether B succeeded. There are four possible outcomes: both succeed, both fail, only A succeeds, or only B succeeds. The cases in which they disagree are especially informative about their difference. If only A succeeds on 10 tasks and only B on 15, the net advantage for B is five tasks out of 100. If the disagreement counts were 0 and 5, the same five-point gap would have a different pattern of trade-offs.

This pairing controls for some shared task difficulty. A difficult bug affects both systems; so does an unusually easy one. Comparing their outcomes on that same bug can remove part of the variation caused by the particular mix of tasks. It does not guarantee a smaller uncertainty interval in every dataset, and it does not correct a biased benchmark. It does make use of information discarded when the individual results are reduced to two separate averages.

Thomas Dietterich's influential paper on statistical tests for comparing supervised classifiers examines several methods and their assumptions. One common test for paired yes-or-no results is McNemar's test. It focuses on the two disagreement counts: cases where A alone is right and cases where B alone is right. Under an appropriate null hypothesis of equal probabilities for the two directions of disagreement, an imbalance gives evidence against equal performance. When disagreements are few, an exact calculation can be preferable to relying on a large-sample approximation.

A statistical test answers a narrower question than many readers assume. A small probability under a null model does not say how large or useful the improvement is, whether a model will win on a different population, or how likely a particular hypothesis is to be true. Report the actual difference and an uncertainty interval as well. Ground Truth's lesson on bootstrap resampling explains one way to estimate uncertainty: resample the task pairs, keeping the two systems' results together, and recompute the difference. This approximates variability from sampling tasks under its assumptions.

The independent unit matters. Suppose the benchmark contains ten repositories, each with ten closely related bugs. Treating all 100 results as independent may give a misleadingly narrow interval if bugs from the same repository share libraries, conventions or causes of failure. A repository-level analysis or a resampling method that keeps each repository's tasks together can better reflect that structure, though only ten repositories still supply limited evidence. Similarly, repeated generations on the same prompt are not automatically independent new prompts. A good report distinguishes variation across tasks from randomness across runs.

The setup must also be fair. Give systems the same task definitions, test conditions, resource limits and opportunity to use tools, or explain a deliberate difference. A cheaper system given fewer attempts may still be the right product comparison, but the result answers a cost-constrained question rather than an unrestricted capability question. Record failures, timeouts, excluded tasks and any changes to the evaluation before looking at results. The lesson on preregistration describes how deciding those rules in advance reduces opportunities to select a favorable result.

Pairing cannot repair a benchmark that is unrepresentative, contaminated or scored incorrectly. A model may have encountered some examples during training; the benchmark contamination lesson covers that concern. Nor does a statistically detectable gain prove that a developer will work faster: a carefully designed user study is needed for that different outcome.

For an applied example, the coding-harness debate contains individual reports about workflow and preference, not a paired evaluation. A stronger experiment could assign the same representative tasks to two harnesses, state model and budget, record task-level outcomes and compare the paired differences. A separate human study could then test whether those results translate into better real-world work. The method makes a comparison clearer, while the choice of tasks and outcomes determines what the comparison actually means.

Ground Truth. (2026, November 8). Paired comparisons: how to tell whether one AI system really beats another. Ground Truth. https://groundtruth.day/learn/paired-comparisons-in-ai-evaluation.html

@misc{groundtruth:paired-comparisons-in-ai-evaluation,
  title  = {Paired comparisons: how to tell whether one AI system really beats another},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {nov},
  url    = {https://groundtruth.day/learn/paired-comparisons-in-ai-evaluation.html}
}

Topics: evaluation · statistics · benchmarks