MCR-Bench: Benchmarking Real-World Multi-Round Code Review with LLMs

Forum topic · 小凯 · 2026-08-29

Summary

Researchers introduce MCR-Bench (arXiv:2508.11368), the first defect state-aware benchmark for realistic multi-round code review with large language models (LLMs). Covering five popular programming languages, MCR-Bench comprises 2,269 real-world multi-round code review tasks, each annotated with fine-grained defect metadata (description, type, severity) and dynamic state labels that track a defect's evolution across interaction rounds. Experiments with mainstream LLMs reveal three key findings: (1) overall capability is limited—LLMs show weak defect detection and defect lifecycle state tracking, with performance degrading notably as interaction rounds increase; (2) performance varies significantly across defect types and severity levels, with semantically complex or low-salience defects more likely to be missed; (3) error analysis identifies distinct drivers of false positives and false negatives, exposing weaknesses such as cross-round temporal misalignment and insufficient long-range memory. MCR-Bench highlights the gap between static single-round evaluation and the iterative nature of real code review.

Paper Overview

Field: NLP Authors: Dewu Zheng, Yanlin Wang, Xiwen Wang arXiv: 2508.11368

Motivation

In real-world software development, code review typically involves iterative interactions between developers and reviewers to improve software quality, making the process costly and time-consuming. Although recent work explores large language models (LLMs) for automated code review, most approaches oversimplify code review into a single-round, static decision task, failing to capture the multi-round interactive nature and complex problem-solving processes inherent in realistic review scenarios.

MCR-Bench

MCR-Bench is the first defect state-aware benchmark designed for realistic multi-round code review. It:

Covers five commonly-used programming languages
Contains 2,269 real-world multi-round code review tasks
Annotates each task with fine-grained defect metadata (description, type, severity)
Includes dynamic state annotations capturing a defect's full evolution trajectory across rounds

Key Findings

1. Limited overall capability: Mainstream LLMs show weak performance on defect detection and defect lifecycle state tracking, with significant degradation as the number of interaction rounds increases. 2. Defect-sensitive performance: Performance varies substantially across defect types and severity levels; semantically complex or low-salience defects are more likely to be missed. 3. Underlying failure mechanisms: In-depth error analysis dissects distinct drivers of false positives and false negatives, revealing critical weaknesses such as cross-round temporal misalignment and insufficient long-range memory.

Why It Matters

MCR-Bench bridges the gap between static single-turn evaluation and the iterative, stateful nature of real-world code review, providing a more realistic yardstick for LLM-based code review systems.

---

Source: arXiv:2508.11368