News · 2026-10-20

whatmodel opens a comparison of lab benchmark claims and independent scores

Jake Manger's whatmodel project reports that model makers' benchmark scores average 6.4% higher than matched independent scores in its collected sample. The project, introduced publicly October 19, exposes its methodology and data rather than treating release charts as directly comparable facts. Its own limitations are central: the pairs are not randomly sampled, testing settings can differ and the statistic is not a universal correction factor for every laboratory or model.

Key facts

• The methodology page reports 611 matched maker-versus-independent score pairs across 180 models and 103 tests.
• The reported mean relative difference is 6.4%; the median is 1.2%.
• Each matched pair receives equal weight, and the reported interval uses resampling by model.
• The primary source is whatmodel's methodology and data, with public source code.

The striking part is not simply that two people can obtain different benchmark scores. It is that release comparisons often place those numbers next to each other without carrying along the instructions, reasoning budgets and software around the model. A score looks compact and objective while representing a much larger experimental setup.

Manger summarizes the observed average as “6.4% higher on average.” That phrase describes a mean of relative differences. For each pair, the calculation takes the maker's score minus the independent score and divides by the independent score. It is not the same as an average gap of 6.4 percentage points, and it does not mean a typical real-world task becomes 6.4% less likely to succeed.

A simple example shows the difference. If a maker reports 60 and an independent test reports 50, the gap is ten points but 20% relative to 50. If another pair has a much smaller independent denominator, an apparently modest point difference can become a large relative difference. The mean and median can consequently tell noticeably different stories. whatmodel publishes both rather than presenting the mean as a complete account.

The reported median of 1.2% is much lower than the mean. That does not invalidate the analysis; it warns against imagining that all pairs cluster near the headline number. The project also says maker scores are higher in 63% of unequal pairs. Some pairs therefore go the other way, and the dataset contains different kinds of tests with different sensitivity to evaluation choices.

The reported 95% interval for the mean is 3.8% to 9.5%, using resampling by model. Grouping at that level is relevant because multiple observations of one model are not necessarily independent evidence about the whole market. But a statistical interval describes uncertainty within an analysis; it cannot remove bias created by the way examples entered the dataset.

The author explicitly says the records are not a random sample. Laboratories choose which results to publish, and independent evaluators choose what they can afford to test. A matched pair can also differ in prompts, harness, reasoning effort or release configuration. Those mechanisms can produce a gap without proving deliberate misrepresentation. They can also make a small claimed lead too fragile to support a strong product conclusion.

An analogy is comparing two cars' fuel economy when one number comes from a laboratory cycle and the other from a mountainous commute. The mismatch is worth investigating, especially if advertisements suppress the testing conditions. But subtracting a fixed percentage from every future claim would not produce a fair road test. The useful response is to recover the conditions, rather than invent a universal adjustment.

This is particularly relevant to agent benchmarks. An agent harness decides what tools exist, how errors are handled and when the model gets another attempt. Reasoning effort changes how much work the model can do before answering. A comparison that labels only the model can conceal those differences even when both reported numbers were measured honestly.

The same issue appears in the MiniMax release discussion, where participants shared the aggregate study alongside task-specific complaints and favorable experiences. The dossier also records Reddit discussion emphasizing selection effects and protocol mismatch. These are reception signals, rather than external replications of the entire dataset.

The project's significance is a reusable accountability workflow: publish pairs, expose the calculation and make it possible to inspect exceptions. Its strongest counterargument comes from its own sampling limitations. Readers should use it to ask better questions of benchmark claims, not to declare every laboratory dishonest or apply 6.4% to an unrelated score. A fair comparison still needs matched conditions, an appropriate test and the humility to distinguish the measured system from the model name printed above it.

Primary source, verified: read the paper →

Ground Truth. (2026, October 20). whatmodel opens a comparison of lab benchmark claims and independent scores. Ground Truth. https://groundtruth.day/news/whatmodel-maker-independent-benchmark-score-gap.html

@misc{groundtruth:whatmodel-maker-independent-benchmark-score-gap,
  title  = {whatmodel opens a comparison of lab benchmark claims and independent scores},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/whatmodel-maker-independent-benchmark-score-gap.html}
}

Topics: benchmarks · evaluation · transparency · open-source · industry