News · 2026-10-16

MathArena uses GPT-6.1 Sol to search 793 papers for mathematical counterexamples

MathArena used GPT-6.1 Sol to search 793 arXiv papers for counterexamples to selected mathematical theorems, reporting major problems in 6% of papers in its earlier-period group. The October 15 report demonstrates a potentially useful screening workflow, while leaving the accuracy of its flags and the representativeness of its sample only partly established.

Key facts

• The sample contains 656 papers from January 2025–July 2026 and 137 from August–September 2026.
• Sol flagged major issues in 6% of the earlier group’s papers and minor issues in a further 30%.
• The researchers report spending about $2,500 in service credits on the audit.
• Primary source: MathArena’s report and methodology, by Jasper Dekoninck, Tim Gehrunger, Kári Rögnvaldsson, Chenhao Sun and Martin Vechev.

A single counterexample can overturn a universal mathematical statement. If someone claims every item in a collection has a property, one legitimate item without it is enough to disprove the claim. That asymmetry makes counterexample search a useful task for an automated assistant: it can sometimes produce a compact object that humans can inspect, rather than asking them to trust a long, fluent proof.

The team’s report is explicitly titled “Searching for errors in mathematical papers with GPT-6.1 Sol.” Its pipeline extracts up to five main reusable theorems from a paper’s source text, asks a coding agent to seek counterexamples, and has the model verify and reclassify what it finds. It also checks whether later paper versions repaired the issue. This is a targeted attempt to falsify selected statements, rather than a complete substitute for refereeing.

The distinction changes how the headline should be read. A theorem can have a false statement because an assumption is missing even when most of the surrounding work remains useful. Conversely, a theorem that survives this search may still contain a faulty proof, lack originality or be important for reasons the system does not assess. Passing a search is evidence about what the search found, not a certificate of correctness. Proof assistants offer a different kind of assurance when a formal statement and proof have actually been checked.

MathArena separates major, minor and edge cases. A major error requires substantially changing or restricting a theorem’s conclusion. A minor error admits a smaller repair that preserves the main contribution. An edge case can be an apparent typo or omitted conventional assumption whose intended meaning is clear. These categories matter because a count that combines them would blur fundamentally broken results with repairable statements.

The most consequential caveat concerns validation. The authors spot-checked 20 errors and found no categorization mistakes. They also asked another model to assess all 75 remaining major-flagged theorems, along with samples from other categories. That model agreed with Sol’s major classification in 78.7% of the relevant cases. Further manual inspection and repeat checks informed the authors’ estimate that a little under 90% of the major findings are correct.

That estimate is not the outcome of an independent mathematician panel checking every finding. Models can share blind spots, and repeated agreement can preserve a mistake. This is a concrete instance of the limits discussed in AI judging: an additional model can help triage a large workload without becoming an unquestionable referee. The report itself treats these checks as estimates of error rates.

The apparent improvement in newer papers also needs restraint. Sol’s major-flag rate fell from 6% before August to 1.5% in August–September. The researchers identify several possible explanations, including changes in the paper population and reduced ability to detect errors in AI-assisted work. That observation does not show that AI caused mathematical error rates to decline, and the report does not establish that its corpus represents all mathematics.

The audit arrives alongside a much broader dispute over automated research. OpenAI’s mathematics announcement and results repository describe a separate collection of machine-generated manuscripts. MathArena did not audit that collection. Rikab Gambhir’s guest essay on Terence Tao’s blog questions who bears the work of checking and understanding a flood of results, while Scott Aaronson’s response defends continued discovery alongside human understanding.

For working researchers, the useful next step is a reproducible triage process: preserve the exact theorem version, inspect the proposed counterexample and contact the paper’s authors before turning a model flag into a public verdict. This study supplies evidence that automated counterexample search can find worthwhile leads cheaply. It does not justify declaring that a fixed fraction of the mathematical literature is wrong.

Primary source, verified: read the paper →

Ground Truth. (2026, October 16). MathArena uses GPT-6.1 Sol to search 793 papers for mathematical counterexamples. Ground Truth. https://groundtruth.day/news/matharena-sol-audit-flags-theorems-not-entire-papers.html

@misc{groundtruth:matharena-sol-audit-flags-theorems-not-entire-papers,
  title  = {MathArena uses GPT-6.1 Sol to search 793 papers for mathematical counterexamples},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/matharena-sol-audit-flags-theorems-not-entire-papers.html}
}

Topics: mathematics · ai-for-science · evaluation · verification