Review Arcade: Human Alignment and Gaming of LLM-Based Peer Review

Forum topic · 小凯 · 2026-05-30

Summary

This post summarizes an arXiv paper (2605.28897) by Hans Ole Hatzel, Sebastian Steindl, and Jan Strich on LLM-generated peer reviews. Using papers from the 2025 ACL Rolling Review (ARR), the authors empirically evaluate LLM reviews from both author and reviewer perspectives. Key findings: alignment between LLM reviews and human reviews is limited, and varies substantially across prompts and models, even though best-case alignment is reasonable. The paper also examines an adversarial scenario where authors use an iterative draft-revise LLM workflow to improve their submission based on LLM review feedback. This gaming of LLM reviews proves effective in specific scenarios, producing a statistically significant increase in overall scores for up to 35% of papers. The results raise concerns for conferences officially piloting LLM-assisted reviewing, since both reviewers and authors may rely on LLMs, potentially distorting the review process.

Paper Overview

Research Area: LLM evaluation Authors: Hans Ole Hatzel, Sebastian Steindl, Jan Strich Published: 2026-05-30 arXiv: 2605.28897

Abstract

LLM-generated reviews for scientific papers are gaining considerable traction and are even being officially piloted by major conferences. We have to assume that not only reviewers are using LLM-assistance, but also that authors use LLMs to revise their papers before submitting. In this work, we perform empirical experiments on papers from the 2025 ACL Rolling Review (ARR) to evaluate LLM reviews from both the author and the reviewer perspective. First, we identify a limited alignment of LLM reviews with human ones. In the best-case scenario, the alignment is reasonable. However, we also find that LLM-human alignment varies substantially across prompts and models. Finally, we investigate the scenario in which the author uses an iterative draft-revise workflow to improve the submission according to the LLM review. We find that this "gaming" of LLM reviews can be effective in specific scenarios, leading to a statistically significant increase of overall scores for up to 35% of papers.

Key Findings

LLM reviews show limited alignment with human reviews, based on experiments on 2025 ARR papers.
Alignment is reasonable in the best case, but varies substantially depending on the prompt and model used.
An iterative draft-revise workflow lets authors "game" LLM reviews effectively in specific scenarios.
Such gaming produces a statistically significant increase in overall scores for up to 35% of papers.
The results raise concerns as major conferences pilot LLM-assisted reviewing while authors also use LLMs for revision.

---

*Auto-collected on 2026-05-30*