StraTA: A Strategic Planning Framework That Cures AI Agents' 'Amnesia' in Long-Horizon Tasks

Forum topic · ✨步子哥 · 2026-05-17

Summary

StraTA is a framework designed to fix the core weakness of LLM-based agents: reactive, step-by-step decision-making that causes them to lose sight of their original goals in long-horizon tasks. The framework introduces two key innovations. First, Hierarchical GRPO extends DeepSeek's Group Relative Policy Optimization to multi-turn decision-making by sampling N global strategies and executing M trajectories under each, comparing candidates at both the strategy level (using the mean reward of the top-delta fraction of trajectories) and the action level, which cleanly separates credit assignment between planning and execution. Second, Farthest Point Sampling (FPS), borrowed from 3D geometry processing, selects semantically diverse strategies via text embeddings to force exploration beyond homogeneous, repetitive plans. The article also offers a practical engineering guide: developers can add a lightweight 'global strategic planner' that generates a natural-language strategy up front and injects it as context at every decision step, without any reinforcement learning training. Reported results are striking: a 7B open-source model with StraTA jumps from 5.3% to 84.2% success on WebShop, reaches 93.1% on ALFWorld, and scores 63.5% on SciWorld, surpassing Claude-class closed models.

StraTA: A Strategic Planning Framework That Cures AI Agents' "Amnesia"

Why LLM Agents "Forget as They Go"

Current LLM-based agents often lose sight of their goals in long-horizon tasks because they use reactive decision-making: each step reacts only to the current observation, with no global plan. This short-sighted strategy fails in complex, multi-step tasks for two reasons:

Aimless exploration: Without a global blueprint, agents wander, take detours, and drift away from the goal (e.g., browsing irrelevant products in a web-shopping task).
Diluted credit assignment: In long trajectories, the final success/failure signal is hard to trace back to early key decisions, so the agent cannot tell which step was right, learns slowly, and repeats mistakes.

Prior methods like ReAct and Reflexion add reasoning traces and reflection, but these remain step-wise reactive planning, not a strategic commitment to the whole trajectory. What agents lack is a "draw the map before departing" global strategy.

Hierarchical GRPO: Extending DeepSeek's Algorithm to Multi-Turn Decisions

StraTA's core innovation is Hierarchical GRPO, extending GRPO (Group Relative Policy Optimization, introduced by DeepSeek) to agent-level multi-turn decision-making, solving the credit assignment problem.

GRPO recap: GRPO generates multiple candidate answers for the same problem, computes advantage values by relative comparison within the group, and updates the policy. This avoids reliance on absolute rewards and mitigates reward sparsity and bias.

Hierarchical innovation: StraTA lifts GRPO to a strategy-action two-level structure:

For each task, the model first samples N distinct global strategies, then executes M full interaction trajectories under each strategy, yielding N×M trajectories.
Strategy-level group: Each strategy is judged by its M trajectories — but not by a simple average. StraTA uses the average reward of the top δ fraction of trajectories, emphasizing a strategy's best execution and avoiding penalizing good plans for noisy early actions.
Action-level group: For each strategy, its M executions form an action-level group compared relatively, teaching the model how to better execute steps under a given strategy.

This hierarchical comparison jointly trains a strategy generator (produce plans that lead to high reward) and an action executor (act optimally under a given plan), cleanly answering "is this plan good?" and "is this action executed well?" separately.

Farthest Point Sampling: Forcing AI Out of Homogeneous Thinking

LLMs tend to generate semantically similar strategies even at high sampling temperatures, limiting exploration. StraTA borrows Farthest Point Sampling (FPS) from 3D geometry processing:

1. Over-sample σ×N candidate strategies. 2. Encode each strategy as a vector with a pretrained embedding model (e.g., Sentence-BERT). 3. Iteratively select: start with the strategy closest to the mean vector, then repeatedly pick the unselected strategy with the minimum maximum cosine similarity to the already-selected set.

The result is N highly diverse strategies, letting the model explore low-probability but potentially superior plans and escape local optima — providing richer material for hierarchical GRPO training.

A Developer's Practical Guide: A "Global Strategic Planner" Without RL

You don't need reinforcement learning to apply StraTA's essence — "draw the map before departing". Steps:

1. Identify long-horizon tasks in your application (e.g., multi-turn complaint resolution, cross-file code changes). 2. Design a global strategic planner: At task start, have a model generate a compact natural-language strategy via well-designed prompts (e.g., "before executing, propose a brief global strategy guiding all subsequent actions"). 3. Inject the strategy into execution: Provide the fixed global strategy as additional context at every decision step, alongside the current observation. 4. Multi-strategy comparison and self-critique (optional): Generate several candidate strategies, have the model critique them, and execute the best one — a lightweight version of hierarchical GRPO. 5. Monitor and iterate: Track cases where the strategy was right but execution failed vs. cases where the strategy itself was flawed; refine prompts accordingly.

This "think first, act second" structure significantly reduces aimless wandering and rework, concentrating compute where it matters.

Conclusion: From Amnesia to Efficient Planning

StraTA shows that explicit global strategies upgrade reactive agents into strategic planners. Hierarchical GRPO co-optimizes plans and actions; FPS forces exploration beyond comfortable thinking; and the engineering pattern can be adopted without any RL. Reported results for a 7B open-source model with StraTA:

| Benchmark | Result | |---|---| | WebShop (before → after) | 5.3% → 84.2% success | | ALFWorld | 93.1% success | | SciWorld | 63.5% score, surpassing Claude-class closed models |

These numbers demonstrate the power of strategic planning for complex decision-making. Expect explicit strategy layers to become standard in future agent designs — just as humans plan before acting on complex tasks. AI's amnesia has a cure: teach it to draw the map before departing, and it will stay on course through even the longest journeys.