Paper Overview
Field: ML Authors: Xin He, Yanlin Wang, Mingwei Liu Published: 2026-09-06 arXiv: 2509.04275
Abstract
Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measure whether generated patches pass functional tests and overlook review-derived acceptance constraints (review constraints) that often influence whether a patch is acceptable in real-world software development.
The authors introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness. SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints. Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation of bug-fixing capability from review constraint compliance.
Key Findings
- The benchmark contains 303 repository-level repair instances spanning 75 open-source Python repositories across diverse software domains.
- Experiments under general coding agent frameworks with four LLM backends of varying capability levels reveal a significant gap between functional success and success under the complete repair specification.
- Of 644 patches that passed functional tests, 221 failed to satisfy the provided review constraints.
- These results demonstrate that functional-only evaluation overestimates an agent's ability to meet the full requirements of repository-level repair tasks.
- arXiv paper: https://arxiv.org/abs/2509.04275
- Replication package (code, data, experimental results): https://github.com/DeepSoftwareAnalytics/SWE-Gate