Harvard study: AI peer reviews beat paid expert reviews of the same papers

Two Harvard researchers ran a head-to-head test of AI against some of the best human peer reviews they could find. Gary King and Steven Worthington of Harvard’s Institute for Quantitative Social Science report the result in a working paper dated 22 October: “No matter how we look at the data, AI wins.”

The setup

The human reviews come from rapidPeer, a Harvard IQSS service that gives faculty paid, double-blind reviews of drafts before journal submission. Reviewers are chosen by field experts, asked for constructive advice of at least 1.5 to 2 pages, and told not to judge fit for a particular journal. The corpus is every eligible submission: 124 papers and 134 human reviews collected from 2019 to 2026, across statistical, mathematical and qualitative work. In a survey, 88% of 74 authors rated their review high or very high quality.

For each paper the authors generated one review with Claude Opus 4.8 at its maximum reasoning setting, under two conditions. In the “matched” one it got the manuscript and the same rapidPeer guidelines, with no tools. In the “tuned” one it got a purpose-built prompt and web search, averaging 12.4 searches per review.

GPT-5.5 judged the pairs, so the reviewer and the judge come from different model families. Each pair was judged in both orders, and a win counted only if the same review won both times. Separately, three former editors of leading journals judged 10 random pairs blind. They did not know one review in each pair was written by AI.

The results

• AI judge: the AI review won all 132 decided pairs overall.
• Human editors: the AI review won 87%, which is 26 of 30 judgments.
• By area: 100% for sources, weaknesses and methods, 99% for figures and tables, and 92% for theory, framing and communication.
• Accuracy: AI reviews cited no fabricated sources. 5.5% of their claims about the manuscript were judged incorrect, against 15.9% for the human reviews.

The paper lists ten robustness checks. Among them: AI still won all 44 pairs of similar length, rewriting human reviews in an AI style did not help them, and a replication with Claude and GPT reviewers and judges from both families found the AI review preferred in 97.1% of cases under the Opus judge and 94.2% under the GPT judge. Without web access, the matched reviews won only 62% on sources.

What it does not show

The authors say the study measures one purpose of review, helping authors improve a paper. It “does not speak to” how editors should decide what to publish. They also did not measure whether revisions actually improved the papers. The paper is a working paper, not yet peer reviewed itself, and the human-editor result rests on 10 pairs.

What they suggest journals consider

The paper’s policy section weighs risks first. AI feedback may push papers and whole fields “toward homogenous arguments and conclusions,” and authors revising against a published prompt may end up optimizing for the reward model rather than the science. The authors then sketch options, from authors running repeated AI reviews before submission, to journals publishing an official review prompt, to editors using an author dossier plus a private AI review, with human reviewers kept for hard cases. They say the final publication decision should stay with a human editor.

They also note that AI use is already common among reviewers. At ICML 2026, which assigned reviewers to AI-use policies at random, 22.5% of surveyed reviewers under a no-AI policy said they used it anyway.