EnvACE Deep Dive: Teaching Agents to 'Rehearse' the World in Their Heads

Forum topic · QianXun · 2026-08-20

Summary

EnvACE (arXiv:2608.06197), a collaboration among Zhejiang University, Shanghai Jiao Tong University, Tencent, CUHK, NUS, Sun Yat-sen University and Central South University, is a new agentic reinforcement learning method that internalizes environment dynamics into the policy itself. Instead of waiting for a real environment or external simulator during training, a shared policy π_θ alternates between two roles: ACT (emitting actions) and REHEARSE (generating the environment's response), with a Role-wise GRPO objective that computes separate baselines per role while updating one set of parameters. At inference, the internalized world model enables private rehearsal: limited-budget imaginary rollouts that don't touch the real environment. EnvACE-8B reaches 32.91% overall across BFCL-v4, τ²-Bench, VitaBench and FinMCP-Bench, edging EnvScaler-8B and AWM-14B, with test-time scaling (N=2 rehearsals) lifting overall to 40.9%. The analysis also debunks hype: the paper never quantitatively validates fidelity of self-generated responses, tests only up to 8B on tool-interaction tasks, and claims reduced—not eliminated—environment dependence. Code: github.com/Within-yao/EnvACE.

> Source: *EnvACE: Internalizing Environment Dynamics via World Rehearsal for Agentic Reinforcement Learning* (arXiv:2608.06197, v1, 2026-08-06) > Institutions: Zhejiang University × Shanghai Jiao Tong University × Tencent × CUHK × NUS × Sun Yat-sen University × Central South University > Code: github.com/Within-yao/EnvACE

Key points

Core idea: Replace external environment interaction during agentic RL training with *World Rehearsal* — the same policy π_θ alternately plays the ACT role (emit an action) and the REHEARSE role (generate the environment's response), then continues reasoning on its self-generated script. This internalizes an agent world model into one set of parameters.
Role-wise GRPO: Because a single trajectory mixes ACT and REHEARSE outputs, naive single-baseline GRPO would let the two signals pollute each other. EnvACE groups outputs by role, computes a per-role baseline (advantage = trajectory reward minus that role's group mean), and jointly updates one shared θ with a clipped GRPO objective. Ablations show shared-parameter beats a split dual-policy variant (36.7% vs 35.5% on τ²-Bench).
Test-time private rehearsal: After training, the model performs N imaginary rollouts before committing to real execution — parallel mode (rehearse N trajectories, aggregate into rehearsal memory) or serial mode (rehearse → refine → rehearse). Rehearsals never touch real environment state. N=2 lifts overall score from 36.7% to 40.9% (+4.2pp).
Training setup: Qwen3-8B backbone (plus 1.7B/4B scaling checks), CM2 dataset, 470 steps, lr 1e-6, batch 16 with 4 rollouts per prompt, KL 1e-4, 12k input / 8k output tokens, max 30 interaction turns, verl framework, 16× NVIDIA H20, Qwen3-30B-A3B as LLM judge.

Results

Overall (four benchmarks): EnvACE-8B 32.91% > AWM-14B 32.54% > EnvScaler-8B 31.92% > base Qwen3-8B 28.48%. A steady but modest lead (+0.99pp over the strongest baseline), not a rout.
τ²-Bench: 36.7%, +3.8pp over EnvScaler-8B and +6.0pp over AWM-14B; standard GRPO reaches only 31.2%, confirming World Rehearsal itself contributes +5.5pp.
BFCL-v4: 46.04% (+2.00pp over base). VitaBench: 16.0%, best among 7B–8B methods.
FinMCP-Bench: best TF1 46.78% (+3.10pp) and tool precision 54.04%, suggesting more disciplined tool use.
Scaling signal: gains from world rehearsal grow with model size (1.7B → 8B), though the paper tests only up to 8B.

Qualitative cases (Fig 7/8)

EnvACE rehearses that a planned API call will fail and fixes parameters (e.g., filling a missing field) before executing; the baseline needs extra recovery steps.
Facing a rule like "economy tickets can't be freely changed," EnvACE foresees the write operation is invalid and switches to a read-only query instead.

Corrections to popular over-readings

"Amazing generalization" — overstated; the lead is small but consistent.
"A cheat-code prophet" — pith.science notes the paper never quantitatively validates that self-generated responses ô_t match real responses o_t. Accurate framing: *an agent that rehearses mentally and thus crashes less*, not a prophet.
"Completely breaks the environment bottleneck" — the paper claims reduced dependence, not elimination; it still executes in a real environment at test time.

Limitations

1. No fidelity check of internalized dynamics — the load-bearing premise is unverified. 2. Only tested to 8B. 3. Only tool-interaction tasks where responses can be approximated in language; numeric simulation, physical control, and high-risk state changes may produce false feedback. 4. Highly stochastic or non-stationary environments may degrade the internalized model. 5. Training cost is real: 16× H20 for 470 steps — it saves environment construction, not compute.

Takeaways for practitioners

1. Dual-role, shared-parameter (ACT/REHEARSE with role-wise baselines) beats splitting into two models. 2. Test-time rehearsal is a new compute–effectiveness knob, better matched to agent error modes than answer re-sampling and voting. 3. First ask: *can your environment feedback be approximated in language?* If yes, world rehearsal is worth trying; if not, stick with real environments or high-fidelity simulators.

Resources

Paper: https://arxiv.org/abs/2608.06197
PDF: https://arxiv.org/pdf/2608.06197
Code: https://github.com/Within-yao/EnvACE