SPADE: Self-Play in Adaptive Synthetic Executable Environments for LLM Training

Forum topic · 小凯 · 2026-08-21

Summary

SPADE (Self-Play in Adaptive Synthetic Executable Environments) is a self-play reinforcement learning framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in those environments. Environments are stateful and multi-turn, covering reasoning problems and multi-step agentic tool use. The agent's regret is estimated as the reward gap with and without privileged prompts; optimizing this regret signal lets the designer target environments at the agent's capability boundary while keeping them feasible. Key design choices include grounding the designer in documents sampled from large pretraining corpora and giving it cumulative environment memory. Scaled to 30B-parameter models, SPADE outperforms the strongest fixed-environment baselines by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and by +5.7 (BFCL-v4 multi-turn) and +13.9 (ACEBench-Agent) in tool-use settings. arXiv: 2608.19197.

SPADE: Self-Play in Adaptive Synthetic Executable Environments

Field: NLP arXiv: 2608.19197

Key Points

Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. Existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales.
SPADE is a self-play RL framework in which a single LLM plays two roles:
Environment Designer: writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface.
Reasoning Agent: learns to act in those environments.
Each environment is stateful and multi-turn (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use.
The reasoning agent's regret is estimated as the reward gap with and without privileged prompts; optimizing this regret signal drives the environment designer to target the agent's capability boundary while keeping environments feasible.

Ablations and Findings

Several components are critical to success: grounding the environment designer in documents sampled from large pretraining corpora, and equipping it with cumulative environment memory.

Results

Scaled to 30B-parameter models, SPADE averages +5.3 over the strongest fixed-environment baseline across eight held-out math, science, code, and reasoning benchmarks.
Tool-use settings: +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent.
In game settings, the advantage over the strongest baseline grows with model scale.

By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.