From Static to Dynamic: Benchmarking Real-World Multi-Round Code Review with MCR-Bench

Forum topic · 小凯 · 2026-08-29

Summary

MCR-Bench (arXiv:2508.11368) is the first defect state-aware benchmark for realistic multi-round code review, covering five commonly-used programming languages with 2,269 real-world multi-round review tasks. Each task is annotated with fine-grained defect metadata (description, type, severity) and dynamic state labels that track how defects evolve across interaction rounds. Unlike prior work that reduces code review to a single-round static decision, MCR-Bench captures the iterative interaction between developers and reviewers. Experiments with mainstream LLMs reveal three findings: overall performance on defect detection and defect lifecycle state tracking is limited and degrades significantly as interaction rounds increase; performance varies notably across defect types and severities, with semantically complex or low-salience defects more likely to be missed; and error analysis identifies cross-round temporal misalignment and insufficient long-range memory as key causes of false positives and false negatives.

论文概要 (Paper Overview)

Research area: NLP Authors: Dewu Zheng, Yanlin Wang, Xiwen Wang Published: 2026-08-28 arXiv: 2508.11368

Introduction

In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work explores large language models (LLMs) for automated code review, most approaches oversimplify code review into a single-round, static decision task, which fails to capture the multi-round interactive nature and the complex problem-solving processes inherent in realistic review scenarios.

MCR-Bench

To bridge this gap, the authors introduce MCR-Bench, the first defect state-aware benchmark designed for realistic multi-round code review:

Covers five commonly-used programming languages
Consists of 2,269 real-world multi-round code review tasks
Each task is annotated with fine-grained defect metadata (e.g., description, type, severity)
Includes dynamic state annotations capturing the full evolution trajectory of defects across rounds

Key Findings

Experiments with mainstream LLMs on MCR-Bench yield several findings:

1. Limited overall capability: Mainstream LLMs show limited performance on defect detection and defect lifecycle state tracking, with performance dropping significantly as the number of interaction rounds increases. 2. Defect-sensitive performance: LLM performance varies considerably across defect types and severity levels; semantically complex or low-salience defects are more likely to be missed. 3. Underlying failure mechanisms: In-depth error analysis dissects the distinct drivers of false positives and false negatives, revealing critical weaknesses such as cross-round temporal misalignment and insufficient long-range memory.

--- *Auto-collected on 2026-08-29.*