reproducibility
NeurIPS papers average six objective mistakes each, up from four News
A study of 2,500 machine learning papers using an automated checker found that the average number of objective mistakes in a NeurIPS paper rose from 3.8 in 2021 to 5.9 in 2025, a 55 percent increase over four years.
An AI replicated 105 ICML orals, and 34 mostly held up News
The research auditing group SAI reviewed all 168 oral papers from ICML 2026, ran full execution-grounded reproductions of 105 of them, and found that only 34 reproduced more than 40 percent of the claims it attempted to check.
An agent's memory records what you said; this compiler records what you did News
A single-author paper argues computer-use agents waste frontier inference re-deriving routines their user already performed, and presents a model-free compiler that turns raw screen capture into a context block 86 times smaller that an agent reads with 98.4 percent accuracy.
Three Days On, Nobody Has Publicly Compiled OpenAI's Ten Proofs News
OpenAI's repository of Lean proofs for ten mathematics results has 434 stars and 39 forks but exactly one commit, no pull requests, and no issues, and no third party has published a build log showing the proofs check.
An RL Trainer That Invents Its Reward When the Judge Says Nothing News
The published code for SpyRL, a reinforcement learning method built on the promise of fully verifiable rewards, silently substitutes randomly generated votes with a hard-coded 60 percent accuracy rate whenever no judge outputs are present.
An open project publishes the recipe for training capable AI agents News
OpenThoughts-Agent releases its full data-curation pipeline, dataset, and experiments -- showing that what an agent learns from matters more than raw size, and letting anyone reproduce it.
SAI ICML 2026 replication results Tool
A browsable record of automated replication attempts against all 168 oral papers from ICML 2026, showing which papers shipped runnable code and how many of each paper's claims actually reproduced. Useful before you build on a result you have only read the abstract of.
Claude Science Tool
An AI workbench that unifies literature search, notebooks, statistics, and cluster compute, and keeps a reproducible record behind every figure. Beta on Mac and Linux.