> You submitted a paper to a top conference. Three months later, you receive four reviews. Three look solid—relevant citations, identified weaknesses, middling scores. But one sentence in the fourth stops you cold: "The proposed method lacks a competitive improvement margin over existing baselines." Grammatically correct, professionally phrased—but a normal person doesn't talk like that. You paste the review into an AI detector and the needle jumps. You start wondering: am I being reviewed by a human, or by a machine? > > This is not hypothetical. In 2026, Żurawicki et al. from Wrocław University of Science and Technology published *PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing*. They had five mainstream LLMs generate eleven thousand reviews covering one thousand real ICLR and NeurIPS papers, then systematically reconciled them against human reviewers. The result is unsettling: machine-written reviews are longer and more complex, but systematically miss critical flaws a human would spot at a glance. Machine scores are more concentrated and more optimistic, and their citation patterns—who they cite to support judgments—follow an entirely different logic from humans.
| Item | Details | |------|---------| | Paper | PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing | | Authors | Krzysztof Żurawicki, Julia Farganus, Arkadiusz Gaweł, Mateusz Bystroński, Tomasz Jan Kajdanowicz | | Institution | Wrocław University of Science and Technology (Poland) | | arXiv ID | 2605.29815 | | Submitted | May 28, 2026 | | Categories | cs.AI, cs.CL | | Scale | 5 LLMs × 1,000 ICLR/NeurIPS papers (2021–2025) = 11,000 reviews | | Core finding | LLM reviewing behavior systematically deviates from humans: lower score variance, higher positivity bias, overconfidence; model-specific citation patterns that don't match human review norms; longer, more complex reviews that frequently miss atomic flaws humans catch |
1. 📝 First, define what "reviewing like a human" means
PRAIB's first move is decomposing "can LLMs review papers?" into measurable dimensions. Prior discussion has been mostly anecdotal—"I had ChatGPT review my paper and it sounded right" is not evidence. PRAIB pushes the conversation from "I feel" to "data says."
It defines three dimensions of review behavior:
- Specificity — does the review cite specific passages, tables, and equations?
- Style — does the language fall within the normal distribution of human reviews?
- Engagement behavior — which references are cited, what scores are given, how confident is the reviewer?
- Was my review written by AI? PRAIB can tell you how LLMs differ as a group, not whether a specific review is human, machine, or hybrid. Individual-level detection would require calibration samples orders of magnitude beyond PRAIB's data. Not feasible today.
- Can prompt engineering fix the biases? The tested prompt strategies failed to converge LLM behavior to the human pattern. That doesn't prove no such prompt exists—only that none of these did.
- Review behavior vs. review quality? PRAIB measures behavior, not quality. Whether LLM reviews or human reviews lead to more accurate acceptance decisions is a different, larger experiment PRAIB deliberately leaves open.
- Cross-domain generalization. All 1,000 papers come from ICLR/NeurIPS. In theoretical CS, bioinformatics, or the humanities, human review baselines themselves differ. PRAIB doesn't discuss whether its "human baseline" carries domain bias.
Together these form a behavioral portrait of human reviewers. PRAIB projects LLM reviews onto it and measures the deviation.
2. 🏭 A production line of 11,000 reviews
The experimental scale is itself a signal: five models spanning closed-source (GPT family) and open-source (Llama family, from 7B to flagship); 1,000 papers drawn from real ICLR/NeurIPS 2021–2025 review data, each with genuine human reviews as a baseline; multiple prompt templates rather than a single instruction. This scale enables analysis at quantiles, distribution shapes, tail behavior, and cross-model consistency—not just "average score differences."
3. 🎯 Three systematic scoring biases
Human scores are naturally contentious—the same paper might draw a 6, 7, 4, and 8. That disagreement is peer review's quality-control mechanism. Machine scores behave differently:
1. Low variance. Machine-generated scores cluster in a much narrower band, rarely giving extreme 3s or 9s—mostly 5–7. Professional-sounding reviews, risk-averse scoring. 2. Positivity bias. Machine scores run higher overall. A paper a human sees as "decent but with hard flaws" becomes "interesting work with some contribution" in an LLM's hands. The likely source is training data: positive evaluation language is more common than harsh criticism, so the model learned the expert's tone without the expert's nitpicking. 3. Overconfidence. When asked how certain it is, the LLM reports systematically higher confidence than humans—and that confidence is uncorrelated with review quality. The machine isn't more accurate; it's just more sure it's accurate.
4. 📚 Citation patterns: who vouches for the machine?
A human reviewer criticizing "insufficient comparison to method X" cites the paper for method X. PRAIB finds LLM citation patterns are not only different from humans, but model-specific. Some models gravitate toward highly-cited classics regardless of topical relevance; some are highly sensitive to prompt wording; some produce citations that look plausible (real paper, right authors, correct year) but are simply the statistically most "fitting" outputs for the context—not associations grounded in the manuscript's content.
Honest caveat: PRAIB did not systematically verify whether the references are correct; it analyzed behavioral features (count, domain distribution, recency). But citation patterns reveal which nodes in a reviewer's knowledge network get activated. For humans, it's "this paper reminds me of pioneering work." For LLMs, it's "which reference titles are statistically most probable in this context." Different cognitive processes, nearly identical surface text.
5. 🔍 Longer, more complex—but missing atomic flaws
Intuition says LLM reviews should be vaguer than human ones. PRAIB's data points the opposite way: LLM reviews are on average longer than human reviews, with more complex sentence structure and more jargon. They look more professional and thorough.
Then PRAIB did something clever: it listed the specific flaws human reviewers identified per paper ("the error bars in Figure 3 are computed wrong," "the ablation lacks strict control of hyperparameter beta," "the comparison with Baseline B is unfair due to different training data sizes") and checked whether LLM reviews mentioned them.
Result: LLMs systematically miss these atomic, specific, actionable flaws—even when objective and obvious. The reviews are longer and prettier, but their "resolution" in examining a paper is far lower than a human's.
A plausible explanation: for long texts, LLMs perform "probabilistic summarization" rather than line-by-line close reading. They capture structure and main arguments, but not the micro-flaws requiring word-level scrutiny. Yet peer review's real value lives exactly at that microscopic level—at least 30% of rejected papers are rejected because a reviewer found, in page 7's third paragraph, a key assumption the authors overlooked.
6. ❓ What honestly can't be said yet
PRAIB is a diagnostic tool, not a cure—and that's a virtue.
7. 🏛️ You hired a grammatically flawless robot to guard the gate
PRAIB recalls a legal principle about courtroom translation: you can't just grab any bilingual person to interpret testimony—you need a certified court translator, because the role carries professional norms (literal versus cultural translation, marking uncertainty) that a merely fluent speaker won't follow. A fluent amateur's testimony reads smoother but transmits less faithful information.
LLM reviews pose the same problem. They read better—longer paragraphs, standardized terminology, a seemingly impartial tone—but reading fluency may come at the cost of informational fidelity. Linguistically, a superior reviewer; cognitively, a text-generation template with statistical preferences.
The paper does not say "LLMs can't review." It says: if you want LLMs reviewing papers, you need to know *where* they differ from humans, in *which direction* the bias points, and what *systematic consequences* it may cause. PRAIB hands the diagnostic toolkit to the community. How to use it—as assistance or replacement, as a screening tool or final judge—is decided not in the paper, but by every editor, area chair, and program committee that picks it up.
| Item | Details | |------|---------| | Paper | PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing | | Authors | Krzysztof Żurawicki, Julia Farganus, Arkadiusz Gaweł, Mateusz Bystroński, Tomasz Jan Kajdanowicz (Wrocław University of Science and Technology) | | arXiv ID | 2605.29815 | | Categories | cs.AI, cs.CL | | Core contributions | (1) First systematic behavioral benchmark of LLM-assisted peer review across specificity, style, and engagement; (2) 11,000 reviews × 1,000 ICLR/NeurIPS papers (2021–2025); (3) three systematic biases: low score variance, positivity, overconfidence; (4) model-specific citation patterns diverging from human logic; (5) longer, more fluent reviews that systematically miss atomic flaws—a trade-off between fluency and fidelity | | Key limitations | No individual-level AI detection; prompt engineering may not eliminate biases (untested beyond PRAIB's strategies); measures behavior, not decision quality; ML-only corpus limits generalization; reference authenticity not systematically verified |
References: 1. Żurawicki et al., "PRAIB: Peer Review AI Benchmark of Behaviour of LLM-Assisted Reviewing", arXiv:2605.29815, 2026. 2. Stelmakh et al., "A Novice-Reviewer Experiment to Address Scarcity of Qualified Reviewers", AAAI 2021. 3. Liang et al., "GPT-4 Generates Review-Like Text but Not Review-Like Judgment", arXiv:2311.12022, 2023. 4. Latona et al., "The AI Reviewers Are Coming: A Survey of LLMs in Peer Review", arXiv:2501.02845, 2025. 5. Goldberg et al., "An Open Review of OpenReview", arXiv:2407.15672, 2024.