Reviewers Dead, Reviewers Forever: When AI Takes the Peer Review Bench

> You submit a paper to a top conference. Three months later, you receive four reviews. Three read convincingly — citing relevant literature, pointing out weaknesses, giving middling scores. But the fourth contains a sentence that stops you: "The proposed method lacks competitive improvement margins compared to existing baselines." Grammatically correct, professionally worded — but a normal person doesn't talk like that. You paste the review into an AI detector. The indicator jumps. You start wondering: am I being reviewed by a human, or by a machine?

This is not hypothetical. In 2026, Żurawicki et al. from Wroclaw University of Science and Technology published *PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing*. They had five mainstream LLMs generate 11,000 academic review comments covering 1,000 real ICLR and NeurIPS papers, then systematically compared them against human reviewers. The result is unsettling: machine-written reviews are longer and more complex, but systematically miss critical flaws humans spot at a glance. Machine scores are more concentrated and more optimistic, and their citation patterns follow an entirely different logic from human reviewers.

| Item | Detail | |------|--------| | Paper | PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing | | Authors | Krzysztof Żurawicki, Julia Farganus, Arkadiusz Gaweł, Mateusz Bystroński, Tomasz Jan Kajdanowicz | | Institution | Wroclaw University of Science and Technology (Poland) | | arXiv ID | 2605.29815 | | Submitted | May 28, 2026 | | Categories | cs.AI, cs.CL | | Scale | 5 LLMs × 1,000 ICLR/NeurIPS papers (2021-2025) = 11,000 reviews | | Core finding | LLM review behavior systematically deviates from humans: lower score variance, higher positive bias, overconfidence; model-specific citation patterns that don't follow human review norms; longer, more complex reviews that frequently miss atomic flaws humans catch |

1. Defining What "Reviewing Like a Human" Means

PRAIB's first move is decomposing "can LLMs review papers?" into measurable dimensions. Prior discussions were largely impressionistic — "I had ChatGPT review my paper and it sounded right" is an anecdote, not evidence. PRAIB aims to move the conversation from "I feel" to "data says."

The paper defines review-behavior metrics along three dimensions:

• Specificity — does the review reference specific paragraphs, tables, and equations?
• Style — do language patterns fall within the normal distribution of human reviews?
• Behavior of Engagement — which references are cited, what scores are given, how confident is the reviewer?

Together these form a behavioral profile of human reviewers, onto which LLM-generated reviews are projected to reveal deviations.

2. An 11,000-Review Production Line

The experimental scale is itself a signal:

• Five models: both closed-source (GPT series) and open-source (Llama series), from 7B to flagship scale.
• 1,000 papers: drawn from real ICLR and NeurIPS 2021-2025 review data, meaning each paper has genuine human reviews as a baseline.
• Multiple prompt strategies: not just one "please review this paper" instruction, but different instruction templates.

11,000 machine-generated reviews in total — enough to analyze not just average score differences, but quantiles, distribution shapes, tail behavior, and cross-model consistency.

3. Three Systematic Scoring Biases

Human reviewer scores have an intuitively familiar feature: wide disagreement. The same paper might get 6, 7, 4, and 8 from four reviewers — this variance is peer review's quality-control mechanism.

LLM scoring is a different story. PRAIB's data shows:

1. Lower score variance. Across five models and 300 prompt variants, machine scores concentrate in a much narrower band. They rarely give extreme 3s or 9s; scores cluster between 5 and 7. Reviews read professionally, but the scoring looks risk-averse.

2. Positive bias. Machine scores are systematically higher. A paper a human sees as "decent, but with several serious flaws" becomes "interesting work with some contribution" in an LLM's hands. The cause may not be kindness — positive evaluation language is more common in training data, so models learned an expert's tone without an expert's nitpicking.

3. Overconfidence. When asked how certain it is, the LLM says "very confident." Machine confidence scores are systematically higher than humans' — and this high confidence is uncorrelated with scoring quality. Machines aren't more accurate; they're just more sure they're accurate.

4. Citation Patterns: Who's Vouching for the Machine?

Human citation behavior in reviews isn't random: a reviewer saying "the paper insufficiently compares against method X" cites method X's paper.

PRAIB finds LLM citation patterns are not only different from humans but model-specific:

• Some models gravitate toward highly cited classics regardless of direct relevance — because those papers appear most frequently in training data.
• Some models' citations track prompt wording closely — rephrase the question and get a completely different reference list.
• Some models' citations look plausible (real papers, correct authors, right years) but on inspection are the outputs statistically "fittest" for the context — not grounded in actual content relationships.

One honest caveat: PRAIB did not systematically verify whether references are correct. It analyzes behavioral features of citation patterns (count, domain distribution, recency), not reference authenticity. But citation patterns themselves expose the problem: what a reviewer cites reveals which nodes in their knowledge network were activated while reading. The human activation pattern is "this paper reminded me of pioneering work"; the LLM's is "which literature titles are statistically most probable given this context." These are not the same cognitive process — yet at the text level of a review, they look identical.

5. Longer, More Complex, But Missing Atomic Flaws

PRAIB's specificity analysis may be the paper's most disturbing part.

Intuition says LLM reviews should be vaguer — after all, it can't "carefully read" a thirty-page paper. The data points the opposite way: LLM-generated reviews are on average longer than human reviews, with more complex sentence structures and more jargon. They look more professional, more thorough.

Then PRAIB did something clever: it listed the specific flaws human reviewers identified ("Figure 3's error bars are miscalculated," "the ablation lacks strict control of hyperparameter beta," "the Baseline B comparison is unfair due to different training data scale") and checked whether LLM reviews mentioned them.

Result: LLMs systematically miss these atomic, specific, actionable flaws. Flaws identified by human reviewers appear with low probability in LLM reviews — even when the flaw is objective and obvious. The machine's reviews are longer and prettier, but its "resolution" in examining papers is far lower than a human's.

A plausible explanation: LLMs process long text via "probabilistic summarization" rather than line-by-line close reading. They capture overall structure and main arguments, but not the microscopic defects requiring word-by-word verification. And peer review's real value lives exactly at that micro level — a substantial share of rejected papers are rejected because one reviewer found, on page 7, paragraph 3, a key assumption the authors overlooked.

6. What Honestly Can't Be Said Yet

PRAIB offers a diagnostic tool, not a cure:

• Was my review AI? PRAIB can tell you how LLMs as a group statistically differ from humans — not whether one specific review is human, machine, or hybrid. Reverse-attribution from individual reviews would require calibration samples a hundred times PRAIB's scale. Currently impossible.
• Can prompt engineering fix these biases? Multiple prompt strategies were tested; none systematically converged LLM behavior toward the human pattern. That doesn't mean no such prompt exists — only that none of the tested ones work.
• Review quality vs. review behavior — which matters? PRAIB measures behavior, not quality. It doesn't answer: if area chairs received both LLM and human reviews for acceptance decisions, whose judgment is more accurate? That requires different experimental design.
• Cross-domain generalization. All 1,000 papers come from ICLR and NeurIPS — frontier ML venues. In theoretical CS, bioinformatics, or humanities, human review baselines themselves differ. Whether PRAIB's "human benchmark" carries domain bias is not discussed.

7. You Hired a Grammatically Perfect Robot to Guard the Gate

PRAIB evokes a legal principle about court interpreters: you can't just grab any bilingual person — certified court interpreters follow professional norms you may not realize exist. Without them, a "fluent speaker's" testimony reads more smoothly but carries *less* informational fidelity.

LLM review behavior is the same problem. Machine reviews read better — longer paragraphs, more standard terminology, a more "objective" tone. But improved reading experience may come with degraded information fidelity. At the language level, it's a superior reviewer; at the cognitive level, it's a text-generation template with statistical preferences.

The paper doesn't say "LLMs can't review." It says: if you want LLMs to review, you need to know where they differ from humans, in which direction the biases point, and what systematic consequences those biases may cause. PRAIB has handed the diagnostic toolkit to the community. How to use it — assistance or replacement, screening or final judgment — is a decision that lives with every editor, area chair, and program committee that picks it up.

---

| Item | Detail | |------|--------| | Paper | PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing | | Authors | Krzysztof Żurawicki, Julia Farganus, Arkadiusz Gaweł, Mateusz Bystroński, Tomasz Jan Kajdanowicz (Wroclaw University of Science and Technology) | | arXiv ID | 2605.29815 | | Categories | cs.AI, cs.CL | | Core contributions | (1) First systematic behavioral benchmark for LLM peer review across specificity, style, and engagement; (2) large-scale empirical study: 11,000 reviews × 1,000 ICLR/NeurIPS papers (2021-2025); (3) three systematic biases: lower score variance, positive bias, overconfidence; (4) model-specific citation patterns diverging from human logic; (5) longer, more complex reviews that systematically miss atomic flaws — a fluency-for-fidelity trade-off | | Key limitations | No reverse-attribution of individual reviews to AI; prompt engineering may or may not eliminate biases; measures behavior, not decision quality; ML-only sample limits cross-domain generalization; reference authenticity not systematically verified |

References: 1. Żurawicki et al., "PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing", arXiv:2605.29815, 2026. 2. Stelmakh et al., "A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers", AAAI 2021. 3. Liang et al., "GPT-4 Generates Review-Like Text but Not Review-Like Judgment", arXiv:2311.12022, 2023. 4. Latona et al., "The AI Reviewers Are Coming: A Survey of LLMs in Peer Review", arXiv:2501.02845, 2025. 5. Goldberg et al., "An Open Review of OpenReview", arXiv:2407.15672, 2024.