News · 2026-10-30

AI-led economics audit separates asserted defects from contested choices

An anonymous AI-led audit of 138 economics works revised its headline findings to distinguish 53 asserted defects from 94 contested methodological choices. The October 29 revision says none of its findings overturns a work’s core result and acknowledges that no domain expert reviewed them. The change substantially narrows the earlier presentation of 147 major problems as headline-changing errors.

Key facts

• The screening report covers 138 works by Emmanuel Saez, Gabriel Zucman, and coauthors.
• Its 147 major findings now split into 53 asserted defects and 94 contested choices.
• The report discloses no domain-expert review and no right of reply for the audited authors.
• Primary sources: the revised audit and its preserved original version.

The story is a useful test of what AI-assisted verification can actually establish. Finding a suspicious assumption is not the same as reproducing an arithmetic error. Reproducing a number is not the same as validating the interpretation attached to it. And two models agreeing with a third model do not provide the independent judgment of an economist checking the evidence.

The project says it asked Claude Opus 5.5 and GPT-6 Astra to examine each work without seeing the other model’s output. The systems checked headline figures against available replication material and original sources, then reviewed citations and assumptions. A third agent tried to refute candidate issues, discard weak ones, and grade the remaining findings. That is the report’s account of its workflow, without an independently reproduced audit of the process.

The distinction resembles building inspection. A screening tool can identify a cracked tile, a load-bearing concern, and a design choice someone dislikes. All three may deserve a note. Calling all three structural failures would make the report harder to use. The revised economics audit’s labels—“defects” and “contested choices”—move toward that necessary separation, although the asserted defects still need checking.

The original page advertised 2,035 problems across the collection and 147 serious enough to change a headline number. The revision says 52 findings concerned limitations or assumptions already disclosed in the works themselves, and another 16 had already been acknowledged or corrected later. These categories are context about the findings, not numbers to add together into a new total. They show why novelty, severity, and correctness require separate accounting.

The revised report also says 72% of 5,110 reproduction checks matched published figures, with 490 additional checks not possible. That is a statement about reproducibility within the report’s attempted checks. It is not a human-confirmation rate for its criticisms. An audit can reproduce a figure exactly and still contest the modeling choice that produced it; it can also fail to reproduce something because information is unavailable rather than because the original figure is wrong.

Attention accelerated after Tyler Cowen’s October 28 post, which praised the effort and argued for reconsidering the research and associated policy arguments. Harvard economist Jason Furman challenged the presentation, particularly the claim that 97% of directional findings weakened the original works. His criticism is methodological: a procedure grading issues by how much they weaken a claim can manufacture a one-way pattern.

The revision addresses that objection in part. It discloses the instrument’s asymmetry and reports similar directional tilts when applied to comparison sets of free-market-associated works and Hoover Institution papers. Those comparisons weaken a claim that the direction is uniquely diagnostic of Saez and Zucman. At the same time, they suggest the screening setup may systematically generate criticisms pointing one way. They do not independently prove that the individual findings are right or wrong.

This is the strongest counterargument to treating the audit as a verdict: the adjudicator was another agent, the models’ agreement rate is not published, and domain expertise has not confirmed the findings. Independent model outputs can share training material and reasoning habits. The relevant background is ensembles and correlated errors, alongside AI-as-judge limitations.

The dossier’s Reddit evidence is only feed metadata, so it cannot establish community consensus. Cowen’s endorsement and Furman’s criticism are attributable positions, not a completed expert resolution. The report itself asks readers to check the linked evidence and treat it as screening.

The practical significance remains substantial. AI can make a large collection of replication questions easier to inspect, and a public revision can improve the quality of that inspection. But the most valuable next step is a documented human check of specific findings and responses from the authors. The revised audit gives readers candidates for that work, rather than 147 established errors or an overturned body of economics.

Primary source, verified: read the paper →

Ground Truth. (2026, October 30). AI-led economics audit separates asserted defects from contested choices. Ground Truth. https://groundtruth.day/news/ai-economics-audit-revises-major-findings.html

@misc{groundtruth:ai-economics-audit-revises-major-findings,
  title  = {AI-led economics audit separates asserted defects from contested choices},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/ai-economics-audit-revises-major-findings.html}
}

Topics: research · replication · economics · evaluation · accountability