When Memory Becomes a Curse: The Triple Trap of Self-Improving AI Agents

*English translation and structured digest of a Chinese forum deep-dive on the paper "On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification" (arXiv:2608.18066, Salesforce AI Research).*

> "We must learn to sail in uncertainty rather than wait for the map to be completed." — Karl Popper

Background: Self-Improving Agents

Self-improving agents maintain a memory bank while learning from an online stream of tasks, extracting reusable "workflows" or "reasoning patterns" after each success:

• AWM (Agent Workflow Memory): extracts full workflows from successful trajectories (e.g., search → filter → compare → order) and injects them into context.
• RBank (ReasoningBank): also learns from failures, extracting general reasoning insights retrieved by a retriever for later tasks.

The paper performs a rigorous autopsy of these two systems and finds three critical fragilities.

Key Findings

1. The Curse of Variance

• Baseline variance is already large: best-worst gaps of 4.4% (WebArena GitLab subset) and 6.67% (SCUBA Service subset) across 3 identical runs.
• Memory mechanisms amplify noise: RBank widens the GitLab gap to 7.8%; the Map domain's standard deviation rises from 1.30% to 3.89% (+200% relative).
• In 17 of 24 (71%) domain-method combinations, variance increased with self-improvement.
• Best-vs-worst runs of the same experiment can differ by up to 10 percentage points.
• Prior work often reported only single runs — equivalent to judging a student by one exam.

2. The Illusion of Task Order (Hidden Curriculum)

• Default WebArena task ID order is implicitly easy → hard (early-task pass rates ~75%, late tasks below 40%).
• Experiments with two shuffled orderings show: RBank gains +1.5% under default order but loses 4.5% under shuffled order.
• Conclusion: previously reported "improvements" were largely an artifact of the hidden curriculum, not genuine self-improvement.

3. The Abyss of Underspecification

• Environment underspecification: memory audits found many plausible-but-infeasible memories, e.g., "call an API," "ask the user to confirm," or "use a Python script" in browser-only environments — dead memories that get repeatedly retrieved.
• Task underspecification: WebArena task 118 ("I have jaw bruxism problem...") intends a shopping-navigation task but the agent interprets it as open-domain medical Q&A, storing irrelevant memories.
• Memory contagion: a lucky success (e.g., using the Haversine formula when a page loaded slowly) gets stored and mechanically reused where it may be wrong.

Mitigation Attempts

| Intervention | Content | Effect | |---|---|---| | +Rub | Provide evaluator rubric and scores to memory construction | Partial relief | | +Env | Provide environment execution feedback (failed clicks, missing elements) | Partial relief | | +PMod | Explicitly forbid memory APIs, external sites, human confirmation | Partial relief | | +All | Combined | Closes 31% of the performance gap |

Even combined, a significant gap remains — the authors note other uncharacterized factors contribute to the fragility (retrieval noise, context-window limits, delayed reward attribution).

Recommendations

For researchers:

• Report multi-run statistics (variance, standard deviation, confidence intervals); single runs are nearly meaningless.
• Control task order — randomization is mandatory, not optional.
• Explicitly model environment constraints and pass them to memory-construction modules.
• Develop task-clarification mechanisms so agents ask rather than guess.

For practitioners — three red lights: 1. Evaluation based on a single run? Results may not reproduce. 2. Tasks in a fixed order? "Improvement" may be an ordering effect. 3. Does the agent know what the environment can and cannot do? Otherwise memory becomes poison.

For the field: at least 3 runs per configuration, stress tests under challenging conditions, periodic human audits of memory banks to prune dead memories, and system designs enabling effective human oversight.

Closing

The paper does not reject self-improving agents; it demands we confront their fragility — the way aircraft engineering accounts for every failure mode. The future lies in systems that recognize their own limitations and proceed cautiously under uncertainty.

References

• Ye, Q., Li, Y., Pruksachatkun, Y., Zhang, J., & Wu, C. S. (2026). On the Fragility of Self-Improving Agents: Variance, Task Order, and Underspecification. *arXiv preprint arXiv:2608.18066*.
• Zhou, S., et al. (2024). Agent Workflow Memory. *NeurIPS 2024*.
• Yu, L., et al. (2024). ReasoningBank. *ICLR 2025*.
• Zhou, S., et al. (2024). WebArena: A Realistic Web Environment for Building Autonomous Agents. *ICLR 2024*.
• Koh, J. Y., et al. (2024). VisualWebArena. *ACL 2024*.