Key Points
- Problem: Reactive LLM agents suffer from exploration collapse and credit-assignment failure on long-horizon tasks, because they never learn to plan globally before acting.
- Core idea: Decouple the trajectory into two jointly trained modules — a *Strategy Generator* that produces a compact natural-language plan, and an *Action Executor* that conditions on the fixed plan plus the current observation at every step.
- Training mechanism:
- Two-level GRPO compares plans at the strategy level and execution variants at the action level, separating "wrong strategy" from "bad execution".
- Per-strategy reward uses the best rollouts rather than the mean, giving a more reliable quality estimate.
- Farthest Point Sampling over strategy embeddings enforces semantic diversity instead of near-duplicate plans.
- A lightweight self-criticism step lets the same LLM reject obviously flawed strategies before rollout.
- Results:
- ALFWorld (text household tasks): 93.1% vs. prior best ~70%.
- WebShop (web shopping): 84.2% vs. 5.3% baseline (~16× lift).
- SciWorld: 63.5%, surpassing frontier closed-source models; SciWorld Lifespan subtask reaches 100%.
- Practical takeaway for engineers: Even without RL compute, the same idea applies in prompt engineering — put a global planner on top of a workflow executor that keeps the plan in context at every step.
- Limitations: Strategy quality depends on the base LLM's common-sense reasoning, self-judgment is imperfect, plan granularity is hard to tune automatically, and hierarchical GRPO is still rollout-expensive.
- Takeaway: The bottleneck of LLM agents is training paradigm, not model size — a 7B model taught to "plan before it acts" can outperform much larger closed-source models on long-horizon tasks.
Source
Paper: *StraTA: Incentivizing Agentic Reinforcement Learning with Strategic Trajectory Abstraction* — Xue et al., arXiv:2605.06642. https://arxiv.org/abs/2605.06642 Code: https://github.com/xxyQwQ/StraTA