News · 2026-10-23

Harvard study finds AI manuscript feedback preferred over selected human reviews

Gary King and Steven Worthington report that AI-generated feedback on research manuscripts was preferred to selected human reviews under their study’s criteria. The October 22 working paper compares reviews for 124 manuscripts and includes a small blinded check by former journal editors. It measures the quality of constructive feedback, rather than whether AI can decide which research deserves publication.

Key facts

• The study includes 124 manuscripts and 134 human reviews from the RapidPeer service.
• Three former editors preferred AI in 87% of judgments across ten randomly selected review pairs.
• Claude Opus 4.8 generated AI reviews; a different model family supplied the main automated judgments.
• Primary source: King and Worthington’s Harvard working paper.

A useful review can be valuable without deciding whether a paper is true. It can find missing explanations, unclear figures, overlooked weaknesses or a confusing argument. That is the work this study addresses. The distinction matters because the headline invites a larger conclusion about peer review than the design supports.

King and Worthington give the paper the forceful title “AI Article Reviews Dominate Human Peer Reviews at Their Best.” The comparison group was deliberately strong. RapidPeer, developed by Harvard’s Institute for Quantitative Social Science, selects reviewers through field experts outside their institutions, pays them and asks for constructive pre-submission feedback. Reviewers received deidentified manuscripts rather than a request to judge fit for a specific journal.

The dataset includes eligible submissions collected from 2019 through 2026. Ten manuscripts received two human reviews, explaining why the number of reviews exceeds the number of papers. The work spans empirical statistical research, formal mathematical models and qualitative research. That variety is useful, but it remains a sample from one service and process. It cannot by itself establish performance across every discipline or the whole journal-refereeing system.

The researchers generated AI feedback in two conditions. A matched condition gave Claude Opus 4.8 the anonymized manuscript and RapidPeer instructions, without external tools. A tuned condition used instructions designed for the AI and web retrieval. The authors say the tuned instructions were written before judging and did not contain the judging rubric. Their supplement provides supporting detail.

The main judge was GPT-5.5, from a different model family. It compared paired reviews in both presentation orders, and only preferences consistent across the two orders counted as wins. Overall judgments considered the reviews themselves; judgments about particular areas also used the manuscript. This design addresses some simple presentation biases. It does not make an automated judge equivalent to an independent expert who verifies a scientific contribution.

The striking human result comes from a much smaller check. Three former editors independently evaluated ten randomly selected pairs, with review order randomized and origin concealed until debriefing. They preferred the AI review in 87% of judgments. That supports the idea that the automated result was not solely an artifact of one model rating another. It also leaves substantial uncertainty about wider human preferences, because the human check covered only ten manuscript pairs.

Think of the study as comparing two sets of editorial notes on a draft. Readers may strongly prefer one because it explains weaknesses more clearly and offers more actionable revisions. That preference is meaningful. It does not establish that following the notes improves the final paper, or that the reviewer would detect a subtly false theorem, a fabricated experiment or an unimportant contribution.

The authors report checks that go beyond style. They say a length-matched subset still favored AI, reducing the concern that longer feedback simply looked more helpful. They also report lower rates of incorrect manuscript claims in AI reviews than in human reviews and no fabricated AI citations in their check. Those remain study-reported findings, rather than independent replications or guarantees for future reviews.

There are important institutional and procedural details. RapidPeer was developed at King’s own institute, a connection readers should know when judging the comparison. The paper reports research-ethics approval and enterprise terms barring model training on user data. Human reviewers could consult supplementary files that the AI did not receive, while the tuned AI condition had web retrieval. These differences belong in an assessment of what was actually compared.

Ground Truth’s lessons on AI as a judge, automation bias and holdout evaluation explain adjacent risks. The sample-complexity lesson explains why the amount of independent evidence matters. No uncertainty calculation can manufacture the larger, representative human evaluation this study did not conduct.

The strongest counterargument is therefore about the target, not whether the reported preference exists: helpful-looking feedback is only one function of scientific review. The authors explicitly did not measure whether revisions improved manuscripts or the consequences of switching review systems. The dossier records repeated social circulation from one account, rather than independent expert endorsement. The defensible conclusion is that AI feedback deserves serious testing as an aid to authors, with publication judgment and scientific validation evaluated separately.

Primary source, verified: read the paper →

Ground Truth. (2026, October 23). Harvard study finds AI manuscript feedback preferred over selected human reviews. Ground Truth. https://groundtruth.day/news/harvard-study-prefers-ai-feedback-over-selected-human-reviews.html

@misc{groundtruth:harvard-study-prefers-ai-feedback-over-selected-human-reviews,
  title  = {Harvard study finds AI manuscript feedback preferred over selected human reviews},
  author = {{Ground Truth}},
  year   = {2026},
  month  = {oct},
  url    = {https://groundtruth.day/news/harvard-study-prefers-ai-feedback-over-selected-human-reviews.html}
}

Topics: science · peer-review · evaluation · anthropic · research