News · 2026-08-10

An AI replicated 105 ICML orals, and 34 mostly held up

The research auditing group SAI has published the results of an automated replication effort covering every oral paper at ICML 2026, and the headline finding is that most of them do not mostly reproduce. SAI reviewed all 168 oral papers, ran full execution-grounded reproductions on 105 of them, and found that only 34 of those 105 reproduced more than 40 percent of the claims it attempted to verify. Just 8 reproduced more than 80 percent.

Key facts

• All 168 ICML 2026 oral papers received a paper-and-code review; SAI's page reports 63 percent replicated at some level.
• 105 papers went through full replication; 34 of those reproduced more than 40 percent of attempted claims, and only 8 exceeded 80 percent.
• 104 of the 105 shipped code that could actually be run, and 92 had at least five separately verifiable claims.
• Primary source: SAI's July 22, 2026 writeup and its ICML results page.

Oral papers are the top of the pile. Of the thousands of submissions to a conference like ICML, a small fraction are accepted and a small fraction of those are selected for oral presentation -- the work the field's own reviewers judged most significant. This is not a survey of marginal papers. It is a survey of the ones the community held up as its best of the year.

What SAI means by replication is narrower and more concrete than the word usually implies. It is not asking whether the idea is good or whether the finding generalizes. It takes the paper's stated claims, obtains the released code, runs it, and checks whether the numbers come out. That is execution-grounded, which is why the code-availability figure is so striking: 104 of 105 papers shipped code that ran. The bottleneck is not open source. It is the distance between code that executes and code that produces the paper's results -- missing hyperparameters, undocumented preprocessing, data that was not released, a seed that mattered more than anyone admitted.

The denominator deserves care, and SAI is transparent about it. The widely shared "a third of top papers" shorthand refers to 34 out of 105, not 34 out of 168. The 105 were not randomly sampled either: SAI estimated the compute each paper would need and worked from cheapest to most expensive. That means the fully replicated subset over-represents work that is inexpensive to rerun, and under-represents large-scale training results, which are both harder to check and arguably where reproducibility problems are worst. The clean statement is: SAI audited all 168 oral papers, fully replicated 105, and a minority of those mostly held up.

SAI is not alone in pointing AI at the scientific record. A separate study, To Err Is Human, takes the complementary approach of hunting objective mistakes inside papers rather than trying to rerun them -- a story we cover separately in Papers now average six objective mistakes each. Between them, the two efforts describe a real shift: automated systems are now being used to audit the literature at a scale no human review process has ever managed, and they are finding a lot. We covered an earlier benchmark in this space in Can an AI agent reproduce real science?.

The strongest counter-argument is that failed replication is not the same as a wrong result, and treating it that way would be its own error. A claim can fail to reproduce because the authors omitted a detail from an appendix, because a dataset license changed, because the replication ran on different hardware, or because SAI's automated pipeline gave up where a determined graduate student would not have. None of those mean the science is false. They mean the paper, as published, is not self-sufficient -- which is a real problem, just a different one from fraud or error.

The honest caveat is that we are grading the grader. An automated replication system makes its own judgment calls about which claims count as verifiable, when a number is close enough, and when to stop. SAI publishes its methodology, which is more than most benchmark efforts do, but nobody has independently audited the auditor. That is worth remembering before treating 34 out of 105 as a settled fact rather than the best current estimate. For background on how these evaluations are constructed and where they go wrong, see how AI is benchmarked.

Primary source, verified: read the paper →

Ground Truth. (2026, August 10). An AI replicated 105 ICML orals, and 34 mostly held up. Ground Truth. https://groundtruth.day/news/an-ai-replicated-105-icml-orals-and-34-mostly-held-up.html

@misc{groundtruth:an-ai-replicated-105-icml-orals-and-34-mostly-held-up,
  title  = {An AI replicated 105 ICML orals, and 34 mostly held up},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {aug},
  url    = {https://groundtruth.day/news/an-ai-replicated-105-icml-orals-and-34-mostly-held-up.html}
}

Topics: reproducibility · evaluation · science · benchmarks · icml · research-integrity