ExecCritic: Learn to Test, Test to Improve for Coding Agents

Forum topic · 小凯 · 2026-09-10

Summary

ExecCritic is a framework that separates test generation from source-code repair to make execution feedback reliable for coding agents. A fail-closed scaffold pairs a Test agent that independently writes repository-native tests with a Repair agent that revises code based on frozen tests' execution feedback. Both roles use Qwen-3.5-35B-A3B as the backbone and are post-trained with role-specific reinforcement learning: the Test agent learns to produce behaviorally valid tests that distinguish correct from incorrect patches, while the Repair agent learns both direct task resolution and feedback-guided revision. On SWE-bench Verified, test quality determines whether feedback helps: with a fixed base Repair agent, tests from the base Test agent lower the resolved rate from a 61.2% no-test baseline to 57.3%, while GPT-5.6-sol tests raise it to 65.3%. Post-training lifts the Qwen Test agent's Base-to-Gold success from 22.2% to 62.2%, and composing both post-trained Qwen agents reaches 72.6% resolved rate, an 11.4-point gain over the no-test baseline without stronger-model or oracle feedback at evaluation. Paper: arXiv 2609.09133.

Paper Overview

Research areas: cs.AI, cs.CL, cs.SE

Authors: Leitian Tao, Baolin Peng, Haorui Wang, Hang Wang, Hao Cheng, Wenlin Yao, Qianhui Wu, Tao Ge, Sharon Li, Jianfeng Gao

arXiv: 2609.09133

Abstract

Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue. Agent-generated tests can encode incomplete or incorrect behavioral targets; when the same trajectory writes both the patch and the test, their errors can agree and create false confidence. We introduce ExecCritic, combining a test-verify-revise scaffold with a role-specific reinforcement learning recipe for training agents within it.

Approach

The scaffold separates test construction from source-code repair:

A Test agent independently generates repository-native tests
A fail-closed harness qualifies and freezes the tests
A Repair agent revises source code from the tests' execution feedback without changing the tests

Both roles use Qwen-3.5-35B-A3B as the backbone and are trained separately:

Learn to Test: the Test agent learns to produce behaviorally valid tests that distinguish correct from incorrect patches.
Test to Improve: the Repair agent learns both direct task resolution and feedback-guided revision.

Results

On SWE-bench Verified, test quality determines whether feedback helps: holding the base Repair agent fixed, tests from the base Test agent reduce the resolved rate from a no-test baseline of 61.2% to 57.3%, whereas tests from GPT-5.6-sol raise it to 65.3%.

Role-specific post-training raises the Qwen Test agent's Base-to-Gold success from 22.2% to 62.2%
Composing the two post-trained Qwen agents reaches 72.6%, an 11.4-point gain over the original no-test baseline
No stronger-model or oracle feedback is needed at evaluation time

---

*Auto-collected on 2026-09-10.*