Zhihui Xie

Excited to release our new work ExecCritic: Learn to Test, Test to Improve for Coding Agents. We introduce a new test-verify-revise agentic scaffold and a role-specific RL recipe to train agents within it. On SWE-bench Verified, ExecCritic improves Qwen3.5-35B-A3B from 61.2% → 72.6%, without stronger-model or Oracle feedback at evaluation time. 🤔 The core problem: Coding agents typically explore the repository, edit the source code, and submit once they believe the patch is ready. But this loop only works if both sides are reliable: the test must capture the right behavior, and the agent must know how to improve from its feedback. When the same trajectory writes both the patch and the test, their errors can agree. A wrong patch may pass a wrong test and appear correct. And even with a useful test, the agent still needs to learn how to interpret execution feedback and turn it into the right source-code revision. Empirically, test quality determines whether feedback helps or hurts. Poor tests can actually make the agent worse, while better tests lead to better repairs. ⚙️ Our approach: EXECCRITIC We separate test construction from source-code repair and train both roles specifically for this new agentic scaffold. Learn to Test A Test agent independently explores the repository and generates a task-specific executable regression test. The harness verifies that it runs and fails cleanly on the buggy repository, then freezes it. issue + repo → generate test → verify on Base repo → freeze verifier Test to Improve A separate Repair agent generates a repair patch, runs it against the fixed verifier, receives execution feedback, and iteratively revises the source code. generate repair patch → run fixed verifier → FAIL → revise repair patch → run verifier → … → PASS → submit The Repair agent cannot modify the test to make its own patch pass. 📈 The results: Learning to test substantially improves test reliability, measured by Base-to-Gold success: Base Qwen-35B: 22.2% + SFT: 39.6% + RL: 62.2% Composing the trained Test and Repair agents reaches 72.6% on SWE-bench Verified, an 11.4 percentage-point improvement over the original 61.2% baseline, without a stronger model or Oracle feedback at evaluation time. 🔍 The broader takeaway: For coding agents, execution feedback alone is not enough. We need agents that can both construct reliable feedback and learn to improve from it. As coding agents become longer-horizon and more autonomous, learning how to test may become just as important as learning how to code. 📄 Paper: arxiv.org/abs/2609.09133 💻 Code: github.com/MSR-Orchard/execc… Led by the amazing @LeitianT during his internship at @MSFTResearch, with huge thanks to our collaborators Baolin Peng, Haorui Wang, Hang Wang, Hao Cheng, Wenlin Yao, Qianhui Wu, Tao Ge, and @JianfengGao0217.