Paper Overview
Research area: ML Authors: Xin He, Yanlin Wang, Mingwei Liu arXiv: 2509.04275
Abstract
Repository-level software engineering benchmarks have significantly advanced the evaluation of coding agents, but existing benchmarks primarily measure whether generated patches pass functional tests and overlook review-derived acceptance constraints (review constraints) that often influence whether a patch is acceptable in real-world software development.
We introduce SWE-Gate, a repository-level benchmark for software engineering agents that explicitly evaluates review constraint compliance alongside functional correctness. SWE-Gate derives review constraints from real pull request review comments and synthesizes repository-level repair instances around these constraints. Each instance provides separate functional and constraint tests, together with non-compliant and gold patches, enabling explicit separation of problem-fixing capability and review constraint compliance.
The benchmark contains 303 repository-level repair instances covering 75 open-source Python repositories across diverse software domains. Experiments under a general coding agent framework, using four LLM backends of different capability levels, reveal a significant gap between functional success and success under the full repair specification: among 644 repairs that passed functional tests, 221 failed to satisfy the provided review constraints. These findings indicate that functional-only evaluation overestimates an agent's ability to meet the full requirements of repository-level repair tasks.
Reproduction package (code, data, and experiment results): https://github.com/DeepSoftwareAnalytics/SWE-Gate