Learn · Intermediate

Simpson’s paradox: when the overall AI score reverses every group result

Simpson’s paradox occurs when an association within each of several groups reverses or disappears after the groups are combined. It matters because a model or intervention can look better overall simply because it was tested on an easier mix of cases, even when it performs worse in every comparable group.

Imagine two AI systems tested on easy and hard questions. Model A answers 90 of 100 easy questions and 2 of 10 hard questions correctly. Model B answers 19 of 20 easy questions and 30 of 100 hard questions correctly. B does better in each group: 95% versus 90% on easy questions and 30% versus 20% on hard ones. Yet A’s overall score is 92 out of 110, or about 84%, while B’s is 49 out of 120, or about 41%. A got many more easy questions; B got many more hard ones.

There is no arithmetic error. An overall accuracy is a weighted average of the group accuracies, and each system’s weights are the proportions of easy and hard cases it receives. A system’s score answers two questions at once: how well it performs in each group and which groups its test contains. Change the mixture and the combined score changes even if the group-specific scores stay exactly the same.

Think of two hospitals treating patients with different levels of illness. A hospital that receives more severe cases can have worse overall survival despite doing better for both mild and severe cases separately. The comparison needs to ask what would happen under a common patient mix. Similarly, comparing AI models requires a common task mix when the goal is to compare their performance on the same population.

A practical correction is standardization. Choose a target mix before looking at the winner, calculate each model’s performance within each relevant group, and weight both models’ group results using that same mix. In the example, assigning half the weight to each difficulty group gives A an accuracy of 55% and B an accuracy of 62.5%. These are hypothetical target-population scores, not the scores each system actually achieved across its original test set.

The target mix matters. A customer whose real workload is almost entirely easy questions may rationally care about that workload’s weighted result. A safety team may care especially about rare hard questions. Neither should pretend that the same overall average answers every decision. Publish the group results and the weights so readers can construct the comparison that fits their own setting. The existing lesson on distribution shift explains why a test mixture can differ from a deployment mixture.

There is an important complication: splitting into groups is not automatically the right causal analysis. If the grouping variable is influenced by the treatment or is a common effect of treatment and outcome, conditioning on it can introduce bias or change the question. Judea Pearl’s introduction to causal inference distinguishes observing groups from estimating the effects of an intervention. The causal graphs lesson shows how confounders, mediators and colliders require different decisions.

Suppose an AI assistant causes some users to attempt harder assignments. Comparing only people who attempted hard assignments could select different kinds of users in the assisted and unassisted groups. The resulting within-group difference may not represent the assistant’s effect on a fixed population. A diagram of plausible causes, a clearly specified intervention and, when feasible, randomized assignment can help decide what should be compared. No table alone identifies which causal explanation is correct.

Simpson’s paradox also concerns more than accuracy. A code-review tool might appear to catch a higher fraction of defects overall because it processes more straightforward changes. A medical classifier’s positive predictive value can change as the share of true cases changes. A routing system’s cost can look lower because its traffic contains fewer hard requests. For each, inspect the relevant denominators and report the group mixture alongside the result. The benchmarking lesson covers further reasons why a single score can conceal important trade-offs.

In practice, begin by defining the population and outcome. Identify groups determined before the compared intervention that plausibly affect difficulty or risk; report each group’s sample size and rate; then compute a shared-weight comparison if it answers the question. Check uncertainty, especially in small groups: two correct answers out of ten do not pin down a stable hard-task rate. Finally, ask whether unmeasured differences remain within groups and whether the chosen mix represents the use case.

The paradox is a warning about an aggregate, not a rule that subgroup results always win. The correct comparison follows from the decision or causal question, the data-generating process and the intended population. When those are explicit, an apparent reversal becomes an understandable consequence of changing weights rather than a mystery.

Ground Truth. (2026, November 7). Simpson’s paradox: when the overall AI score reverses every group result. Ground Truth. https://groundtruth.day/learn/simpsons-paradox-and-weighted-averages.html

@misc{groundtruth:simpsons-paradox-and-weighted-averages,
  title  = {Simpson’s paradox: when the overall AI score reverses every group result},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {nov},
  url    = {https://groundtruth.day/learn/simpsons-paradox-and-weighted-averages.html}
}

Topics: statistics · evaluation · causal-inference · benchmarks