GRPO Throws Away Terminal Feedback — ECHO Turns It Into a Free World Model
> Source: ECHO: Terminal Agents Learn World Models for Free, arXiv:2605.24517
---
1. Introduction: A Gold Mine Hidden in the Terminal
A CLI agent works simply: the model issues a command, the terminal executes it, and returns stdout, errors, files, logs, and traces. This chain of feedback records the true consequences of every action the model takes.
Here's the problem: how does standard reinforcement learning train agents?
It only cares about outcomes — task succeeded, reward +1; failed, reward 0. The entire intermediate terminal output is treated as noise and ignored.
The ECHO team says: that's wrong. Terminal feedback isn't noise — it's a dense supervision signal. Every environment response token tells the model "you just did this, and here's how the world responded." Throwing this information away effectively cuts your training data in half.
---
2. The Core Problem: How Much Information Does GRPO Waste?
GRPO (Group Relative Policy Optimization) is the mainstream RL method for agent training today. Its logic: sample a group of trajectories, compute advantages from sparse task-success rewards, and update the policy.
This pipeline has a hidden waste:
- Action tokens are trained by the policy gradient loss
- Environment tokens (everything the terminal returns) are simply ignored
- Action decisions (standard GRPO)
- Environment prediction (auxiliary cross-entropy)
- Traditional world models require extra training, extra data, extra architectures
- ECHO gets this for free — reusing the existing GRPO compute pipeline, since terminal feedback was already there
- Action prediction requires understanding "what I want to do"
- Environment prediction requires understanding "how the world responds"
- Combined, the model gains a more complete understanding of the action-consequence causal chain
- Validated only in terminal environments (CLI agents); applicability to GUI or physical environments is unverified
- Prediction accuracy of environment tokens depends on how structured the terminal output is — free-form natural language logs may be harder to predict than structured JSON
- The auxiliary loss weight requires tuning; optimal values may differ across environments
- Extremely complex environment dynamics (multi-process races, network latency) may exceed current models' prediction capacity
- If you're training CLI agents with GRPO, adding ECHO is nearly zero cost
- If you lack expert data, ECHO can fill a big chunk of the supervision gap
- If you need agents to self-evolve in unfamiliar environments, ECHO's environment prediction offers the possibility of verifier-free improvement
- ECHO: Terminal Agents Learn World Models for Free, arXiv:2605.24517, https://arxiv.org/abs/2605.24517
- Authors: Vaishnavi Shrivastava, Piero Kauffmann, Ahmed Awadallah, Dimitris Papailiopoulos
Failed trajectories are especially wasted. GRPO can barely extract policy gradient signal from them — the task failed, advantages approach 0, and the model doesn't know where things went wrong. Yet these failed trajectories contain rich environmental response information: which commands cause errors? Which output formats indicate a problem in the previous step?
ECHO's core insight: environment observations are not context, they are supervision signals.
---
3. The ECHO Method: One Loss Function, Two Signals
ECHO stands for Environment Cross-entropy Hybrid Objective. The design is minimal: add an auxiliary cross-entropy loss on top of the standard GRPO policy gradient loss.
What does the auxiliary loss do? It trains the policy to predict the environment observation tokens produced by its own actions.
Concretely: the model outputs a sequence of commands; the terminal returns a sequence of responses. ECHO requires the model, while outputting commands, to also predict what the terminal will return. It's not predicting the next action (that's GRPO's job) — it's predicting the environment feedback triggered by the action.
The elegance of this design is compute reuse: ECHO shares the same forward pass as GRPO. No extra sampling, no extra rollouts. The terminal feedback is already there; nobody just asked the model to predict it before.
One loss function, two things learned:
Same network, two objectives, zero extra compute cost.
---
4. Experiments: Doubled Performance, No Experts Needed
Results on the TerminalBench-2.0 benchmark are direct:
| Model | GRPO | ECHO | Gain | |------|------|------|------| | Qwen3-8B | 2.70% | 5.17% | 91% | | Qwen3-14B | 5.17% | 10.79% | 109% |
pass@1 roughly doubles. Note: this is achieved with zero extra compute cost.
Even more striking is the comparison with expert SFT: starting from base Qwen3-8B, ECHO matches the "expert SFT then GRPO" combined pipeline without any expert demonstrations. ECHO alone recovers roughly half of the benefit that expert SFT initialization provides.
What does this mean? High-quality supervised data (expert SFT) is genuinely useful, but ECHO shows that environment feedback itself can supply a large amount of equivalent supervision. It doesn't replace expert data — it reduces dependence on it.
---
5. World Model Capability: Predicting Unseen Trajectories
ECHO doesn't just make the policy stronger — it also learns to predict environment dynamics.
Tested on held-out rollouts (trajectories the model did not generate): ECHO-trained policies significantly reduce cross-entropy on environment tokens, while pure GRPO barely changes. This shows ECHO isn't "memorizing its own generated trajectories" — it's extracting general regularities of the environment, working like a world model.
How this differs from traditional world models:
---
6. Verifier-Free Self-Improvement: Self-Evolution on OOD Tasks
In some settings, ECHO's environment prediction loss alone achieves verifier-free self-improvement.
What does that mean? On unseen OOD tasks, the model can keep improving purely through environment interaction — no external verifier, no reward signal. How?
The model's ability to predict environment responses lets it "anticipate" the consequences of its actions. In OOD environments, this anticipation helps it avoid dead ends and choose more sensible command sequences. Environment prediction becomes an implicit exploration strategy, reducing trial-and-error costs in unfamiliar settings.
This addresses a long-standing RL problem: sparse rewards plus poor generalization. ECHO turns environment responses into dense supervision, letting the model learn from feedback at every step rather than receiving a single 0/1 signal at task end.
---
7. Technical Depth: Why the Auxiliary Loss Doesn't Interfere
A common worry: will adding an auxiliary loss interfere with policy gradient optimization?
ECHO's design avoids this. The auxiliary loss is computed only on environment tokens; the policy gradient only on action tokens. The two losses act on different token positions and don't compete for gradients. The shared Transformer backbone learns both kinds of representations, but the output heads have clear division of labor.
Looking deeper, the two tasks are actually complementary:
This also explains why ECHO can match expert SFT + GRPO — expert SFT gives "humans know what to do," while ECHO gives "the world tells you what happens if you do this." Two supervision sources, same destination.
---
8. Limitations and Boundaries
The paper candidly notes several limitations:
---
9. Practical Takeaways: Every Agent Developer Should Try ECHO
ECHO's lesson is so simple it almost feels like cheating: you're already collecting terminal feedback — you just haven't let it participate in training.
For agent product builders:
Implementation cost: one auxiliary cross-entropy loss. Essentially a one-line change.
---
10. Conclusion: A Free Lunch — Because You've Been Throwing It Away
The paper's title says "Learn World Models for Free" — free not because there's no cost, but because you've already paid the cost.
During GRPO training, every rollout contains complete terminal feedback. ECHO simply says: don't throw these tokens away — let them participate in training too.
It's like a reinforcement-learning version of recycling — the terminal output you thought was garbage is actually a gold mine. If a model can predict "what will the terminal return if I run this command," it has already grasped the basic regularities of the environment.
> "Environment observations are not context, they are supervision signals."
---
References