Navigating the Long Horizon: A Survey Deep-Dive on Agent Architectures and RL for Long-Horizon Sequential Decision-Making

> Paper: Navigating the Long Horizon: A Comprehensive Survey of Agent Architectures and Reinforcement Learning for Extended Sequential Decision-Making > Author: Deli Chen (auto-generated by the Deli AutoResearch framework) > Models: DeepSeek-V4-Pro (text and reasoning) + GPT-Image-2 (figures) > Version: V4.1 (June 4, 2026)

Key points

1. The finale of the Deli AutoResearch trilogy

This is the third autonomously generated survey from the Deli AutoResearch framework, forming a complete "AI self-awareness" trilogy:

| Paper | Core question | Positioning | |---|---|---| | From Copilots to Colleagues | What can AI do? | Capability definition | | Never Stop Learning | How does AI retain and grow capabilities? | Learning mechanisms | | Navigating the Long Horizon | How does AI apply capabilities reliably in complex tasks? | System integration |

2. Why long-horizon tasks remain AI's fatal weakness

The paper defines long-horizon sequential decision-making as tasks requiring dozens to thousands of interdependent steps, and identifies six core challenges:

• C1 Credit assignment: a bug written on line 5 fails at line 500—how to localize it? (e.g., code debugging)
• C2 Exploration: no reward for 100+ steps—how to know the direction is right? (e.g., finding diamonds in Minecraft)
• C3 Compositional generalization: knowing chopping and stirring, can the agent compose "make a salad"?
• C4 Catastrophic forgetting: after learning web operations, does it forget file operations?
• C5 Grounding: how does "open the drawer" become concrete motor commands? (robotics)
• C6 Scalability: does planning 1000 steps become 100× harder than 100 steps? (combinatorial explosion)

Key finding: no single method solves all six challenges—methods each break through only a few walls of this six-dimensional "capability wall."

3. The exponential decay boundary

Theorem 1 (Exponential decay bound): with per-step error rate ε over H steps:

\[P_{success} = (1-\varepsilon)^H\]
• Achieving H > 200 requires per-step reliability > 99.65%
• Current SOTA per-step accuracy on WebArena is ~90–95%, implying 30-step success rates of only ~21–42%
• This matches actual WebArena success rates of 20–30%

| Paradigm | Decay form | Key mechanism | |---|---|---| | Reactive agents | Pure exponential e^(-εH) | No hierarchical decomposition | | Hierarchical methods | Reduced effective H: e^(-εH/k) | Temporal abstraction turns 100 steps into 10×10 | | Search + verification | Lower per-step ε | Multi-path attempts + process verification |

Conjecture (contextual degradation): per-step error rate of Transformer agents may grow over time, causing super-exponential decay—collapsing faster than pure exponentials.

4. Five method families: strengths and blind spots

The survey covers 280+ papers across five families:

1. Hierarchical planning (Options, MAXQ, DEPS, Voyager, ADaPT): decompose long tasks into subtasks. Good at C3/C5/C6; weak at C1 (error propagation across subtasks). 2. Reactive agents (ReAct, Reflexion, SWE-Agent, Devin): Plan→Act→Observe→Reflect loops. Good at C4 (memory externalization) and C2 (trial and error); weak at C6 (pure exponential decay). 3. Search-based planning (ToT, LATS, PRM, Best-of-N): generate and select among candidate paths. Good at C1 (process verification) and C2; weak at C6 (search-space explosion). 4. Reinforcement learning (HIRO, PPO, GRPO, DreamerV3): learn policies from reward signals. Good at C1 (reward shaping) and C2 (intrinsic rewards); weak at C5 (needs massive environment interaction). 5. World models (Dreamer, MuZero, JEPA): simulate environments in imagination. Good at C2 and C6; weak at C5 (sim-to-real gap).

Core conclusion of the gap matrix: no method achieves full coverage, but hybrid architectures are the most promising path.

5. Memory architectures: the lifeline of long-horizon tasks

| Memory type | Effective horizon | Performance profile | |---|---|---| | No memory | H ≈ 50 | Sharp degradation | | Working memory (short-term) | H ≈ 100 | Linear decay | | Full episodic memory + retrieval | H ≈ 200 | Slow decay |

Key insight: memory externalization (storing information in external databases rather than model parameters) is the key to breaking horizon limits—consistent with the LoRA-isolation strategy in *Never Stop Learning*. Representative systems: MemGPT (OS-style virtual memory), MemoryBank (time-aware long-term memory), Voyager (executable skill library).

6. Current SOTA vs. humans

| Domain | Benchmark | SOTA | Human | Gap | |---|---|---|---|---| | Software engineering | SWE-bench Verified | 76% | ~95% | 19% | | Web navigation | WebArena | 58% | ~90% | 32% | | Desktop control | OSWorld | 23% | 72% | 49% | | Scientific research | MLE-bench | 75% (medals) | 100% | 25% | | Mobile apps | AndroidWorld | 35% | ~85% | 50% |

Weakest link: GUI-heavy tasks (desktop/mobile), with 49–50% gaps—visual understanding and cross-app coordination are the biggest bottlenecks. Most mature: software engineering (76%), partly thanks to deterministic verifiers (compile/test).

7. Critical reflections

1. Theorem 1's simplifying assumptions treat step errors as independent and irreversible; real errors are often correlated (cascade failures) and partially recoverable, so actual decay may be faster or slower than exponential. 2. Hybrid architecture pitfalls: coordination overhead between components, unclear optimal decomposition granularity, and verifier accuracy all affect overall performance. 3. The contextual degradation conjecture is unproven; if true, Transformers have intrinsic horizon limits regardless of parameter scale. 4. The meta-question of AI-generated papers: as with the prior two papers, citation authenticity and reproducibility remain concerns—and there is a deeper paradox: if AI can generate a survey on long-horizon decision-making, does it understand its own limits as a "long-horizon task" (6 days to generate the paper)?

8. Implications for future research

• Short term (1–2 years): empirical validation of hybrid combinations; memory/retrieval optimization; "good enough" automatic verifiers for open domains.
• Mid term (3–5 years): verifying the contextual degradation conjecture; adaptive self-learned decomposition; unified cross-modal long-horizon frameworks.
• Long term (5+ years): theoretical limits of bootstrapped hybrid architectures for unbounded H; autonomous long-horizon scientific discovery; the boundary of human-AI collaboration as H→∞.

Conclusion: end or beginning of the recursion?

The three papers form a complete self-referential system: capability definition → learning mechanism → system integration. The next question may be: "How do I improve the way I improve myself, to better apply my capabilities in long-horizon tasks?"—the singularity of recursion, where the object of improvement becomes improvement itself. Deli AutoResearch has produced not merely three surveys but a recursive structure of self-awareness, attempting to understand itself—including the act of understanding itself.

---

References

Chen, D. (2026). Navigating the Long Horizon: A Comprehensive Survey of Agent Architectures and Reinforcement Learning for Extended Sequential Decision-Making. *Generated by Deli AutoResearch framework using DeepSeek-V4-Pro and GPT-Image-2*. V4.1.

Chen, D. (2026). From Copilots to Colleagues: A Survey of Autonomous Research Agents. *Generated by Deli AutoResearch*.

Chen, D. (2026). Never Stop Learning: A Survey of Continual Learning and Self-Iteration in Large Language Models. *Generated by Deli AutoResearch*. V5.