ExecCritic: Learn to Test, Test to Improve for Coding Agents

Forum topic · 小凯 · 2026-09-10

Summary

ExecCritic is a framework that improves coding agents by separating test construction from source-code repair. A Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code based on execution feedback without altering the tests. Both roles use Qwen-3.5-35B-A3B as the backbone and are trained separately with role-specific reinforcement learning: the Test agent learns to produce behaviorally valid tests distinguishing correct from incorrect patches, while the Repair agent learns direct task resolution and feedback-guided revision. On SWE-bench Verified, test quality determines whether feedback helps: tests from a base Test agent lower the resolved rate from a 61.2% no-test baseline to 57.3%, while GPT-5.6-sol tests raise it to 65.3%. Post-training lifts the Qwen Test agent's Base-to-Gold success from 22.2% to 62.2%, and combining both post-trained Qwen agents reaches 72.6%, an 11.4-point gain over the baseline without stronger-model or oracle feedback.

Paper Overview

Research areas: cs.AI, cs.CL, cs.SE
Authors: Leitian Tao, Baolin Peng, Haorui Wang, Hang Wang, Hao Cheng, Wenlin Yao, Qianhui Wu, Tao Ge, Sharon Li, Jianfeng Gao
Published: 2026-09-08
arXiv: 2609.09133

Abstract

Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue. Agent-generated tests can encode incomplete or incorrect behavioral targets; when the same trajectory writes both the patch and the test, their errors can agree and create false confidence.

The authors introduce ExecCritic, combining a test–verify–revise scaffold with a role-specific reinforcement learning recipe for training agents within it. The scaffold separates test construction from source-code repair:

A Test agent independently generates repository-native tests.
A fail-closed harness qualifies and freezes them.
A Repair agent revises source code from execution feedback without changing the tests.

Both roles use Qwen-3.5-35B-A3B as the backbone and are trained separately. In Learn to Test, the Test agent learns to produce behaviorally valid tests that distinguish correct from incorrect patches. In Test to Improve, the Repair agent learns both direct task resolution and feedback-guided revision.

Key Results (SWE-bench Verified)

Test quality determines whether feedback helps: holding the base Repair agent fixed, tests from the base Test agent reduce the resolved rate from a no-test baseline of 61.2% to 57.3%, whereas tests from GPT-5.6-sol raise it to 65.3%.
Role-specific post-training raises the Qwen Test agent's Base-to-Gold success from 22.2% to 62.2%.
Composing the two post-trained Qwen agents reaches 72.6%, an 11.4-point gain over the original no-test baseline — without stronger-model or oracle feedback at evaluation time.

---

*Auto-collected on 2026-09-10.*