MCR-Bench: A Benchmark for Real-World Multi-Round Code Review with LLMs

Forum topic · 小凯 · 2026-08-30

Summary

Researchers introduce MCR-Bench, the first defect state-aware benchmark designed to evaluate large language models (LLMs) on realistic multi-round code review. Unlike prior work that reduces code review to single-round static decision tasks, MCR-Bench captures the iterative, interactive nature of real review processes. The benchmark spans five commonly used programming languages and contains 2,269 real-world multi-round code review tasks, each annotated with fine-grained defect information and cross-round state labels. Experiments with mainstream LLMs reveal three key findings: (1) overall capability is limited, with performance degrading significantly as the number of interaction rounds increases; (2) performance varies substantially across defect types and severity levels, indicating sensitivity to defect characteristics; and (3) error analysis uncovers underlying failure mechanisms such as cross-round temporal misalignment and insufficient long-range memory. The benchmark highlights the gap between static evaluation setups and real-world review workflows. Paper: arXiv:2608.27442.

Paper Overview

Research Area: NLP Authors: Dewu Zheng, Yanlin Wang, Xiwen Wang, Kefeng Duan, Hongyu Zhang, Xilin Liu, Yuchi Ma, Zibin Zheng arXiv: 2608.27442

Abstract

In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work explores large language models (LLMs) for automated code review, most approaches oversimplify code review into a single-round, static decision task, which fails to capture the multi-round interactive nature and the complex problem-solving processes inherent in realistic review scenarios.

To bridge this gap, the authors introduce MCR-Bench, the first defect state-aware benchmark designed for realistic multi-round code review.

Key Characteristics

Covers five commonly-used programming languages
Contains 2,269 real-world multi-round code review tasks
Each task is annotated with fine-grained defect information and cross-round state labels

Key Findings

Extensive experiments with mainstream LLMs reveal:

1. Limited overall capability: Performance degrades significantly as the number of interaction rounds increases. 2. Defect-sensitive performance: LLM performance varies substantially across different defect types and severity levels. 3. Underlying failure mechanisms: Error analysis reveals critical weaknesses such as cross-round temporal misalignment and insufficient long-range memory.