SPADE: Self-Play in Adaptive Synthetic Executable Environments
- Field: NLP
- Authors: Bo Liu, Simon Yu, Yiding Jiang, Ao Qu, Andrew Zhao, Zichen Liu, Junsu Kim, Zijian Zhou, Seungone Kim, Tongzheng Ren, Mickel Liu, Hanfei Yu, Zhaorun Chen, Weiyan Shi, Paul Pu Liang, Luke Zettlemoyer, Yejin Choi, Natasha Jaques
- arXiv: 2608.19197
- Existing training environment pools for language agents (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales.
- SPADE is a self-play RL framework where a single LLM acts as both an Environment Designer (writing complete, long-horizon environments as executable code with a Gym-style
reset()/step()interface) and a Reasoning Agent that learns to act in them. - Each environment is stateful and multi-turn, with state transitions, reward functions, and verification code, so one interface covers reasoning problems and multi-step agentic tool use.
- The agent's regret is estimated via its reward gap with and without a privileged prompt; optimizing this regret drives the designer to target the agent's capability frontier while keeping environments feasible.
- Critical components for success: grounding the environment designer in documents sampled from large pretraining corpora, and giving it cumulative environment memory.
- Scaled to 30B-parameter models, SPADE averages +5.3 over the strongest fixed-environment baseline across eight held-out math, science, code, and reasoning benchmarks.
- Tool-use gains: +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; in game settings the advantage over the strongest baseline grows with model scale.
- By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.
Abstract (original)
Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. T...