> arXiv: 2608.06197 > Team: Zhejiang University, National University of Singapore, Sun Yat-sen University, Central South University, Tencent, CUHK, Shanghai Jiao Tong University > Code: https://github.com/Within-yao/EnvACE
1. The Problem: Why Is Agent Training So Expensive?
Imagine training an AI assistant that can book flights, track packages, and call APIs. Traditional approaches have drawbacks:
Path A: Real environment interaction — the agent actually calls travel APIs and queries databases. This burns API costs, risks data privacy, and gives uncontrollable environment states. Worse, real environments cannot provide diverse feedback for edge cases like failed rebookings or malformed order IDs.
Path B: External simulators — another LLM simulates the environment. But simulators hallucinate and must themselves be calibrated against real environments; you end up training a coupled Agent+Simulator system.
EnvACE's core insight: instead of depending on an external environment, let the agent learn to "imagine" how the environment will respond.
2. Core Innovation: World Rehearsal
EnvACE proposes an aggressive training paradigm—World Rehearsal—summarized in one line:
> One model plays both the "actor" (ACT) and the "theater" (REHEARSE), conducting both roles within its own parameters.
2.1 Dual-Role Mechanism
In each interaction turn, the model alternates between two roles:
- ACT role: given interaction history h_t, generate a tool call a_t
- REHEARSE role: given (h_t, a_t), generate the environment response ô_t
- ACT outputs tool calls (structured JSON, function names, arguments)
- REHEARSE outputs environment responses (simulated API returns, error messages, user feedback)
- Combining World Rehearsal with external verifiers: rehearse first, then filter plausible trajectories with a lightweight verifier.
- Multi-agent rehearsal: agents rehearse each other's responses (game-theoretic scenarios).
- Integration with tool learning: quickly adapt when new tools are introduced via rehearsal.
The history is updated: h_{t+1} = h_t ⊕ (a_t, ô_t)
Key point: the environment response comes not from outside, but from the model's own "imagination." The whole training trajectory is self-unfolded—the model converses with itself.
2.2 Parameter Sharing = Internalized World Model
EnvACE does not use two separate models; a single policy network shares all parameters for ACT and REHEARSE.
This means when the REHEARSE role learns "order IDs must start with #", the ACT role immediately benefits—they share the same θ.
Ablations confirm this: splitting ACT and REHEARSE into independent policies (Per-role Policy) drops tau2-Bench from 36.7% to 35.5%. Parameter sharing yields a 1.2% absolute gain—evidence that knowledge is genuinely internalized.
3. Training: Role-wise GRPO
EnvACE optimizes with GRPO (Group Relative Policy Optimization), with one key modification—role-separated baselines.
3.1 Why Separate Baselines?
ACT and REHEARSE have completely different output distributions:
Mixing them into one baseline means a REHEARSE "normal response" could be dragged down by an ACT "erroneous call," and vice versa.
3.2 Role-wise Advantage
For each role r ∈ {ACT, REHEARSE}, EnvACE computes the average reward of that role's outputs as the baseline and normalizes advantages per role. The ACT and REHEARSE outputs of the same trajectory receive the same trajectory reward R_i, but their advantages are computed against their role-specific baselines. Both roles thus jointly update shared parameters without interfering with each other.
4. Test Time: Private Rehearsal
This is EnvACE's most compelling feature—a trained model can rehearse "in its head" before actually executing.
4.1 Two Rehearsal Modes
Parallel mode: generate N independent imagined trajectories simultaneously, then aggregate experience.
Sequential mode: the n-th rehearsal sees the previous n-1 trajectories and self-evaluation feedback, enabling targeted corrections—like discovering "economy class can't be rebooked" in rehearsal one, then checking rules first in rehearsal two.
4.2 Rehearsal Memory → Actual Execution
All rehearsal results are aggregated into a rehearsal memory m_x, and the ACT role performs one Committed Execution based on it—the only genuine interaction with the external environment.
Key point: rehearsal is private—it triggers no real API calls and mutates no real database state. The AI can "trial and error" in its head as many times as needed, committing only the best plan.
5. Results: Leading Across Four Benchmarks
5.1 Main Results (BFCL-v4, tau2-Bench, VitaBench)
| Method | BFCL-v4 | tau2-Bench | VitaBench | Overall | |--------|---------|------------|-----------|---------| | EnvScaler-8B | 47.07 | 32.9 | 15.8 | 31.92 | | AWM-14B | 47.32 | 30.7 | 19.6 | 32.54 | | EnvACE-8B | 46.04 | 36.7 | 16.0 | 32.91 |
EnvACE-8B's overall score of 32.91% beats all environment-scaling baselines, including the 14B AWM. Especially on tau2-Bench (realistic airline, retail, telecom scenarios), EnvACE's 36.7% substantially leads EnvScaler-8B's 32.9%—demonstrating World Rehearsal's advantage in stateful, multi-turn interactions.
5.2 FinMCP-Bench (Financial Agents)
EnvACE achieves 46.78% on TF1 with the highest tool precision of 54.04%. Financial scenarios demand extreme precision—one wrong trade call costs real money—so EnvACE's high precision embodies the value of "looking before leaping."
5.3 Ablations: Does World Rehearsal Really Help?
Versus standard GRPO: 36.7% vs 31.2% on tau2-Bench, a 5.5% gain.
Versus Per-role Policy: shared parameters 36.7% vs separated 35.5%, a 1.2% gain.
Training dynamics: steady improvement from 30.0% at step 50 to 36.7% at step 470, showing World Rehearsal provides sustained, scalable learning signal.
6. Deeper Analysis: Why Does World Rehearsal Work?
6.1 From "Predicting" to "Playing"
Traditional methods make the model *predict* environment responses (an auxiliary objective); EnvACE makes it *play* the environment (a core mechanism). Prediction is passive hindsight—"I saw action a, I guess outcome o." Playing is active causal internalization—"if I do a, the environment responds this way, so I'd better do b." Internalizing this causality grants counterfactual reasoning—the hallmark of a world model.
6.2 From "Data-Hungry" to "Bootstrapped"
The bottleneck of agent RL is environment interaction data. EnvACE's rehearsal is essentially bootstrapping—the model generates training data from its own imagination. This resembles human learning: we mentally play out chess moves before committing, and simulate an opponent's reactions before negotiating.
6.3 A New Dimension of Test-Time Scaling
Test-Time Scaling (TTS) usually means "compute more reasoning steps." EnvACE's Private Rehearsal adds a new dimension: not a longer chain of thought, but exploring multiple possible interaction paths in parallel before execution—reminiscent of AlphaGo's Monte Carlo tree search: play a thousand games in your mind before placing a stone.
7. Limitations and Outlook
Limitations: 1. World Rehearsal assumes the model has sufficient prior knowledge to simulate the environment—for brand-new domains (e.g., a novel SaaS API), initial rehearsal quality may be low. 2. Rehearsal quality is capped by model capacity—the 1.7B EnvACE performs far worse than the 8B. 3. The paper does not deeply address error accumulation in rehearsal (if step 3 is wrong, later steps may cascade).
Outlook:
8. One-Sentence Summary
> EnvACE teaches LLMs to "run it through in their heads" before acting—not as an extended chain of thought, but as an internalized world model. When AI knows both how to act and how the world will respond, it truly takes the key step from "tool" to "intelligent agent."
---
*Reference: Xu et al., "EnvACE: World Rehearsal for Scalable LLM Agent Training", arXiv:2608.06197, 2026.*