News · 2026-08-10

NeurIPS papers average six objective mistakes each, up from four

A study titled "To Err Is Human: Systematic Quantification of Errors in Published AI Papers via LLM Analysis" reports that the average number of objective mistakes in a published NeurIPS paper rose from 3.8 in 2021 to 5.9 in 2025 -- a 55.3 percent increase. The researchers used an automated checker built on GPT-5 to scan 2,500 papers across three major venues -- NeurIPS, ICLR, and the Transactions on Machine Learning Research journal -- looking only for errors that can be checked rather than judged. Human experts confirmed 83.2 percent of a sampled subset of the flags.

Key facts

• Average objective mistakes per NeurIPS paper: 3.8 in 2021, rising to 5.9 in 2025, a 55.3 percent increase.
• 2,500 papers sampled across ICLR, NeurIPS, and TMLR.
• Human experts reviewed 316 flagged issues and confirmed 263 as real, for 83.2 percent precision.
• Primary source: arXiv:2512.05925.

The design choice that makes this study worth taking seriously is what it refuses to measure. Most attempts to assess paper quality end up measuring taste -- is this novel, is it important, is it well written -- and taste does not compare across years or reviewers. This checker looks only for mistakes with a right answer: an algebra step that does not follow, a derivation with a dropped term, a number in the text that contradicts the number in the table, a figure that does not match its caption. Those are the paper-writing equivalent of typos in a bank statement. They do not require a judgment about whether the idea is good, only about whether the arithmetic is right.

That restriction is what turns a fuzzy complaint -- everyone says paper quality is slipping -- into a measurable trend. From 3.8 to 5.9 errors per paper is roughly two extra checkable mistakes making it through peer review in a typical accepted paper, over four years in which submission volumes at these venues grew enormously. The study does not claim to know why. But the obvious candidates are all structural: more submissions per reviewer, shorter review cycles, more papers written under deadline pressure, and increasingly, more text drafted with AI assistance that is fluent enough to slip past a tired reader.

The verification step is the part that separates this from a press release. Automated error detection is exactly the kind of task where a language model will confidently invent problems that are not there. The authors checked: human experts examined 316 flagged issues and confirmed 263 as genuine, giving 83.2 percent precision. That is a good number for this kind of task, and it also means about one flag in six was wrong. The honest reading of "5.9 mistakes per paper" is therefore "roughly five, plus or minus," not a precise census. Our lesson on using a model as a judge covers why that measured precision matters more than the raw counts, and why an unvalidated automated grader is worth very little.

This lands alongside a separate effort by the group SAI, which tried to rerun the code for every oral paper at ICML 2026 and found that most did not mostly reproduce -- covered in An AI replicated 105 ICML orals, and 34 mostly held up. The two approaches are complementary in a useful way. SAI asks whether the results come out when you run the code. This study asks whether the paper is internally consistent on its own terms. A paper can pass one and fail the other, and the fields' quality problem is clearly not a single thing.

The strongest counter-argument is that error count is not error weight. A mislabeled axis and a broken proof both count as one objective mistake, and they are not remotely equivalent. A rising count could reflect longer papers with more appendices and more surface area to be wrong on, rather than sloppier science. The study's own framing is careful about this -- it quantifies mistakes, it does not rank papers -- and the more alarming interpretation belongs to the coverage, not the authors.

The honest caveat is one the study cannot escape: an AI is now auditing a literature that is increasingly written with AI help, and the same fluency that lets a model draft a plausible derivation is what lets a flawed derivation read as correct to a human reviewer. That is a loop worth watching. It is also, for now, the only method anyone has that scales to 2,500 papers, and the alternative -- trusting that peer review is catching this -- is exactly the assumption the numbers undercut.

Primary source, verified: read the paper → (arXiv 2512.05925)

Ground Truth. (2026, August 10). NeurIPS papers average six objective mistakes each, up from four. Ground Truth. https://groundtruth.day/news/neurips-papers-average-six-objective-mistakes-each-up-from-four.html

@misc{groundtruth:neurips-papers-average-six-objective-mistakes-each-up-from-four,
  title  = {NeurIPS papers average six objective mistakes each, up from four},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/neurips-papers-average-six-objective-mistakes-each-up-from-four.html}
}

Topics: reproducibility · peer-review · evaluation · llm-as-a-judge · science · research-integrity