EnvACE Explained: World Rehearsal Enables LLM Agents to Train Without Real Environments

Forum topic · 小凯 · 2026-08-20

Summary

EnvACE is a training framework from a multi-institution team (Zhejiang University, NUS, Sun Yat-sen University, Central South University, Tencent, CUHK, SJTU) that lets LLM agents learn without expensive real-environment interaction or external simulators. Its core idea, World Rehearsal, uses a single parameter-shared policy that alternates between two roles: ACT (generating tool calls) and REHEARSE (imagining environment responses), so the agent self-unfolds training trajectories and internalizes a world model. Training uses a modified GRPO with role-wise baselines so the two output types do not interfere. At test time, Private Rehearsal lets the model pre-run parallel or sequential imagined trajectories, aggregate them into a rehearsal memory, and commit only once to the real environment. On benchmarks, EnvACE-8B scores 32.91% overall across BFCL-v4, tau2-Bench, and VitaBench, beating EnvScaler-8B and the 14B AWM baseline, with 36.7% on tau2-Bench and a 5.5% gain over standard GRPO. Code is available on GitHub.

EnvACE: World Rehearsal for Scalable LLM Agent Training

> arXiv: 2608.06197 > Team: Zhejiang University, National University of Singapore, Sun Yat-sen University, Central South University, Tencent, CUHK, Shanghai Jiao Tong University > Code: https://github.com/Within-yao/EnvACE

The Problem: Why Agent Training Is So Expensive

Training an agent that books flights, tracks packages, or calls APIs typically requires one of two costly paths:

Real-environment interaction: burning API credits, risking data privacy, uncontrollable state, and poor coverage of edge cases (failed rebookings, malformed order IDs).
External simulators: a second LLM simulating environment responses, introducing hallucinations and requiring calibration against the real environment — you end up training a coupled Agent+Simulator system.

EnvACE's insight: instead of depending on an external environment, let the agent learn to imagine how the environment will respond.

Core Innovation: World Rehearsal

A single large model plays both the actor and the theater:

ACT role: given interaction history h_t, generate tool call a_t
REHEARSE role: given (h_t, a_t), generate imagined environment response ô_t
Update history: h_{t+1} = h_t ⊕ (a_t, ô_t)

The trajectory is entirely self-unfolded — the model talks to itself. Crucially, ACT and REHEARSE share all parameters of one policy network, so knowledge learned by one role (e.g., "order IDs must start with #") is immediately available to the other. Ablations confirm this: splitting into per-role policies drops τ²-Bench from 36.7% to 35.5% (+1.2% from parameter sharing).

Training: Role-wise GRPO

ACT outputs (structured tool calls) and REHEARSE outputs (simulated API responses) have very different distributions. EnvACE computes a separate baseline per role: both roles in a trajectory receive the same trajectory reward R_i, but advantages are normalized against each role's own baseline. This allows joint updates of shared parameters without cross-role interference.

Test Time: Private Rehearsal

Before committing to the real environment, the trained model rehearses privately:

Parallel mode: N independent imagined trajectories generated simultaneously.
Sequential mode: rehearsal n sees previous trajectories and self-evaluations, enabling targeted correction.

All rehearsals are aggregated into a rehearsal memory m_x, and ACT then performs one Committed Execution — the only real environment interaction. Rehearsals never trigger real API calls or modify real state.

Results

| Method | BFCL-v4 | τ²-Bench | VitaBench | Overall | |--------|---------|----------|-----------|---------| | EnvScaler-8B | 47.07 | 32.9 | 15.8 | 31.92 | | AWM-14B | 47.32 | 30.7 | 19.6 | 32.54 | | EnvACE-8B | 46.04 | 36.7 | 16.0 | 32.91 |

EnvACE-8B outperforms all environment-scaling baselines, including the 14B AWM, and leads strongly on τ²-Bench's stateful, multi-turn service scenarios.
On FinMCP-Bench (financial agents): TF1 score 46.78%, tool precision 54.04% (highest overall).
vs. standard GRPO on τ²-Bench: 36.7% vs 31.2% (+5.5%).
Training dynamics: steady rise from 30.0% (step 50) to 36.7% (step 470), indicating sustained, scalable learning signal.

Why It Works

1. From predicting to playing: prediction is passive post-hoc guessing; playing the environment actively internalizes causality, enabling counterfactual reasoning — a hallmark of world models. 2. Bootstrapping over data hunger: rehearsal turns the model's own imagination into training data, sidestepping the environment-interaction bottleneck of agent RL. 3. A new dimension of test-time scaling: rather than lengthening one reasoning chain, Private Rehearsal explores multiple interaction paths before acting — conceptually similar to AlphaGo's tree search.

Limitations and Outlook

Limitations: 1. Rehearsal assumes sufficient prior knowledge to simulate the environment; quality may be low for brand-new APIs/domains. 2. Rehearsal quality is bounded by model capacity (EnvACE-1.7B underperforms 8B). 3. Error accumulation within rehearsal trajectories is not deeply analyzed.

Future directions: combining rehearsal with lightweight external verifiers; multi-agent rehearsal for game-theoretic scenarios; rapid adaptation to new tools via rehearsal.

One-Sentence Takeaway

EnvACE teaches LLMs to "mentally walk through actions before executing them" — not as longer reasoning chains, but as an internalized world model. When an AI knows both how to act and how the world will respond, it takes a key step from tool to agent.

*Reference: Xu et al., "EnvACE: World Rehearsal for Scalable LLM Agent Training", arXiv:2608.06197, 2026.*