ECHO: Turning Terminal Feedback into Free World Models for GRPO-Trained CLI Agents
> Source: ECHO: Terminal Agents Learn World Models for Free, arXiv:2605.24517
1. The Goldmine Hiding in the Terminal
A CLI agent works simply: the model issues a command, the terminal executes it, and returns stdout, errors, files, logs, and traces. This stream of feedback records the true consequences of every action the model takes.
The problem: standard RL for agents only cares about the outcome — +1 reward on task success, 0 on failure. The long stream of terminal output in between is treated as noise and ignored.
The ECHO team argues this is wrong. Terminal feedback is not noise — it is a dense supervision signal. Every environment response token tells the model "here is what you just did, and here is how the world responded." Discarding this information throws away more than half the training signal.
2. How Much Information Does GRPO Waste?
GRPO (Group Relative Policy Optimization) is the mainstream RL method for agent training. It samples a group of trajectories and computes advantages from sparse task success/failure rewards.
This pipeline has a hidden waste:
- Action tokens are trained by the policy gradient loss
- Environment tokens (everything the terminal returns) are ignored entirely
- Action decisions (standard GRPO)
- Environment prediction (auxiliary cross-entropy)
- Validated only in terminal (CLI) environments; GUI or physical environments are untested
- Prediction accuracy depends on how structured terminal output is — natural-language logs may be harder to predict than structured JSON
- The auxiliary loss weight needs tuning per environment
- Highly complex dynamics (multi-process races, network latency) may exceed current models' prediction capacity
- Training CLI agents with GRPO? Adding ECHO is near-zero cost
- Short on expert data? ECHO fills a large chunk of the supervision gap
- Need agents that self-improve in unfamiliar environments? ECHO's environment prediction enables verifier-free improvement
- ECHO: Terminal Agents Learn World Models for Free, arXiv:2605.24517, https://arxiv.org/abs/2605.24517
- Authors: Vaishnavi Shrivastava, Piero Kauffmann, Ahmed Awadallah, Dimitris Papailiopoulos
Failed trajectories are especially wasted. GRPO extracts almost no policy gradient signal from them — the advantage is near zero, so the model never learns *where* things went wrong. Yet these failures contain rich environmental information: which commands cause errors, which output formats indicate a problem upstream.
ECHO's core insight: environment observations are not context — they are supervision.
3. The Method: One Objective, Two Signals
ECHO stands for Environment Cross-entropy Hybrid Objective. The design is minimal: on top of the standard GRPO policy gradient loss, add an auxiliary cross-entropy loss that trains the policy to predict the environment observation tokens generated by its own actions.
Notably, this exploits compute reuse: ECHO shares the same forward pass as GRPO, requiring no extra sampling or rollouts. The terminal feedback was already there — nobody had simply asked the model to predict it.
One loss function learns two things simultaneously:
Same network, two objectives, zero additional compute cost.
4. Experiments: Doubled Performance Without Experts
Results on the TerminalBench-2.0 benchmark are direct:
| Model | GRPO | ECHO | Gain | |-------|------|------|------| | Qwen3-8B | 2.70% | 5.17% | 91% | | Qwen3-14B | 5.17% | 10.79% | 109% |
pass@1 roughly doubles, with zero extra compute cost.
Even more striking: starting from base Qwen3-8B, ECHO with no expert demonstrations matches the combined "expert SFT + GRPO" pipeline. ECHO alone recovers roughly half of the expert-SFT initialization benefit. Expert data is useful, but ECHO shows environment feedback itself supplies much equivalent supervision — reducing dependence on expert data rather than replacing it.
5. World-Model Capability: Predicting Unseen Trajectories
ECHO doesn't just make the policy stronger — it learns to predict environment dynamics. On held-out rollouts (trajectories the model didn't generate), ECHO-trained policies significantly reduce cross-entropy on environment tokens, while pure GRPO barely changes. This shows ECHO is not memorizing its own trajectories but extracting general environment regularities — functioning like a world model.
The difference from traditional world models: those need extra training, data, and architectures; ECHO gets one for free by reusing existing GRPO compute.
6. Verifier-Free Self-Improvement on OOD Tasks
In some settings, ECHO's environment-prediction loss alone achieves verifier-free self-improvement: on unseen out-of-distribution tasks, the model keeps improving through interaction alone, with no external verifier or reward signal. Its ability to predict environment responses lets it anticipate action consequences, avoid dead ends, and choose more sensible command sequences — turning environment prediction into an implicit exploration strategy.
This addresses the classic RL pain point of sparse rewards + poor generalization by converting environment responses into dense supervision available at every step.
7. Why the Auxiliary Loss Doesn't Interfere
A common worry: does adding an auxiliary loss disturb policy gradient optimization? ECHO avoids this — the auxiliary loss is computed only on environment tokens, the policy gradient only on action tokens. The two losses act on disjoint token positions, so they don't fight over gradients. The shared transformer backbone learns both kinds of representations with clearly divided output heads.
Deeper down, the two tasks are complementary: action prediction requires understanding "what do I want to do," environment prediction requires understanding "how does the world respond." Together they give the model a more complete grasp of the action-consequence causal chain. Expert SFT says "here is how a human does it"; ECHO says "here is what the world tells you will happen." Two supervision sources, converging on the same result.
8. Limitations
The paper candidly notes:
9. Practical Takeaway
ECHO's implication is almost cheating-level simple: you are already collecting terminal feedback — just stop excluding it from training.
For agent builders:
Implementation cost: one auxiliary cross-entropy loss.
10. Closing Thoughts
The paper's title says "Learn World Models for Free" — free not because there's no cost, but because you already paid it. Every GRPO rollout contains complete terminal feedback. ECHO just says: don't throw those tokens away.
> "Environment observations are not context — they are supervision."
---
Reference