MCR-Bench: A Defect State-Aware Benchmark for Real-World Multi-Round Code Review with LLMs

Forum topic · 小凯 · 2026-08-30

Summary

MCR-Bench is the first defect state-aware benchmark for evaluating large language models (LLMs) on realistic multi-round code review tasks. Introduced by researchers including Dewu Zheng, Yanlin Wang, Hongyu Zhang, and Zibin Zheng, the benchmark covers five commonly-used programming languages and contains 2,269 real-world multi-round code review tasks, each annotated with fine-grained defect information and cross-round state labels. Unlike prior approaches that oversimplify code review as a single-round, static decision task, MCR-Bench captures the iterative, interactive nature of real review workflows between developers and reviewers. Extensive experiments with mainstream LLMs reveal three key findings: (1) overall capability is limited, with performance degrading significantly as interaction rounds increase; (2) performance varies markedly across defect types and severity levels; and (3) error analysis uncovers underlying failure mechanisms such as cross-round temporal misalignment and insufficient long-range memory. The benchmark provides a foundation for future research on automating multi-round code review with LLMs. Paper: arXiv:2608.27442.

Overview

Field: NLP Authors: Dewu Zheng, Yanlin Wang, Xiwen Wang, Kefeng Duan, Hongyu Zhang, Xilin Liu, Yuchi Ma, Zibin Zheng arXiv: 2608.27442

Abstract

In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work explores large language models (LLMs) for automated code review, most approaches oversimplify code review into a single-round, static decision task, which fails to capture the multi-round interactive nature and the complex problem-solving processes inherent in realistic review scenarios.

To bridge this gap, the authors introduce MCR-Bench, the first defect state-aware benchmark designed for realistic multi-round code review. MCR-Bench covers five commonly-used programming languages and consists of 2,269 real-world multi-round code review tasks, each annotated with fine-grained defect information and cross-round state labels.

Key Findings

Through extensive experiments with mainstream LLMs, the paper identifies:

1. Limited overall capability: LLM performance degrades significantly as the number of interaction rounds increases. 2. Defect-sensitive performance: LLM performance varies markedly across different defect types and severity levels. 3. Underlying failure mechanisms: Error analysis reveals critical weaknesses such as cross-round temporal misalignment and insufficient long-range memory.