Learn · Intermediate

Item response theory: separating model ability from question difficulty

Item response theory is a family of statistical models that estimates test-taker ability and question difficulty from patterns of answers. Applied to AI, it helps separate a model’s performance from the particular mix of easy and hard questions in a benchmark. Its estimates are useful measurements under explicit assumptions, rather than universal intelligence scores.

A raw percentage hides two moving parts. A model can score highly because it is capable, because the test is easy for its strengths, or both. A question can have a high failure rate because it is difficult or because the tested models are weak at that skill. Measuring either side without considering the other makes comparisons harder to interpret.

Georg Rasch developed a foundational model of this relationship in educational measurement. Its simplest form gives each test taker an ability value and each question a difficulty value. The probability of a correct answer depends on the difference between them. When ability equals difficulty, the model predicts a 50% chance of success.

Imagine a climbing wall with routes of different difficulty. Counting completed routes gives a score, but a wall dominated by beginner routes cannot tell two expert climbers apart. Likewise, an extremely hard wall may reveal little about the difference between beginners. Estimating both climber ability and route difficulty creates a shared scale, provided the climbing model fits what people actually do.

For the one-parameter logistic model, success follows a smooth S-shaped curve. An ability value one unit above a question’s difficulty corresponds to about a 73% predicted chance of answering correctly; two units above gives about 88%. These numbers illustrate the mathematical model. They are not empirical guarantees that an AI with a particular leaderboard position will answer a real question at those rates.

Researchers fit the parameters using a matrix of responses: models down one side, questions across the other, and correct or incorrect answers in the cells. Repeated attempts can provide more information about stochastic systems. The fitting process looks for ability and difficulty values that make the observed pattern plausible. It does not discover a question’s difficulty by reading its wording alone.

There is an identification problem: adding the same constant to every ability and difficulty leaves the predicted probabilities unchanged. Researchers therefore fix a reference for the scale. This is why a reported difficulty value is meaningful relative to a calibration procedure, rather than an absolute quantity like a distance in metres. Comparing independently fitted scales requires additional linking assumptions or shared reference items.

The recent paper Your Benchmark Is an Empirical Measure Over Difficulty uses a one-parameter model to examine the difficulty mix of AI tests. Its authors distinguish mismatch from gaps. Mismatch means a test’s questions put weight in the wrong places for a chosen target. A gap means there are no questions in a region where improvement might occur. Reweighting cannot recover evidence from an empty region.

This is a more specific tool than the broad concerns in how AI is benchmarked. A test can be uncontaminated and carefully scored yet still contain mostly questions that all leading models pass. Its average will then be insensitive to the differences a buyer needs to measure. Conversely, a test dominated by impossible tasks can hide progress among weaker systems.

The method also has important failure modes. A one-dimensional scale assumes a coherent ability ordering. Real AI systems can have crossed strengths: one retrieves facts well, another solves spatial problems. A single ability value may fit that pattern poorly. More flexible item response models can give questions different discrimination or model additional dimensions, but extra parameters need enough data and do not eliminate questions about what the test represents.

Difficulty estimates can also move when the calibration panel changes. Questions solved by memorization may behave differently for models with different training exposure. Tool access, prompts, and scoring rules change the response process. The apparent precision of a fitted number should never replace checks for benchmark contamination, uncertainty, or validity on the intended task.

Percy Liang and colleagues’ HELM framework complements this perspective by emphasizing multiple scenarios and metrics. A difficulty scale addresses one part of measurement; it does not replace cost, reliability, fairness, or domain coverage. Similarly, Artificial Analysis’s methodology makes evaluation composition relevant when comparing changing benchmark versions.

To use the idea well, define the workload first, estimate difficulty from a disclosed model panel, inspect how well the response model fits, and report uncertainty and empty regions. A uniform easy-to-hard mix can be a diagnostic reference, but a real application may need mostly routine tasks with a few rare, consequential ones. Item response theory helps make that choice visible. It does not decide which questions society or a customer should value.

Ground Truth. (2026, October 30). Item response theory: separating model ability from question difficulty. Ground Truth. https://groundtruth.day/learn/item-response-theory-ability-and-question-difficulty.html

@misc{groundtruth:item-response-theory-ability-and-question-difficulty,
  title  = {Item response theory: separating model ability from question difficulty},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/learn/item-response-theory-ability-and-question-difficulty.html}
}

Topics: evaluation · item-response-theory · measurement · benchmarks