Memory Backfire: When LLM Experience Summarization Turns Into Poison

Forum topic · 小凯 · 2026-05-24

Summary

A forum post on zhichai.net analyzes a 2026 paper by Dylan Zhang et al. (UIUC, Tsinghua, UChicago, UWashington) titled "Useful Memories Become Faulty When Continuously Updated by LLMs" (arXiv: 2605.12978). The study shows that consolidated memory — where LLM agents rewrite past trajectories into reusable text lessons — degrades with continuous updates. On ARC-AGI, GPT-5.4 scored 100% without memory, but only 52.6% after 10 rounds of per-task memory updates, while one-shot static consolidation kept 94.7%. The authors identify three failure mechanisms: misgrouping, overgeneralization, and overfitting to narrow streams. Grounding the findings in complementary learning systems theory, they argue current agents perform schema formation without episodic retention. Controlled experiments show an Auto mode (retain raw trajectories, consolidate selectively) with an episodic buffer reaches 37.8% cumulative success versus 23.8% for forced abstraction. Engineering recommendations: treat raw trajectories as first-class evidence, gate consolidation explicitly, and support memory rollback.

Paper: *Useful Memories Become Faulty When Continuously Updated by LLMs* Authors: Dylan Zhang et al. (UIUC, Tsinghua, UChicago, UWashington) arXiv: https://arxiv.org/abs/2605.12978 | May 25, 2026

Key points

Counterintuitive result: GPT-5.4 without memory solved 19/19 ARC-AGI problems (100%). The same model with an "auto-summarize experience into memory" system dropped to 52.6% accuracy after 10 rounds of updates. The memory itself was degrading.
Non-monotonic memory utility: On ScienceWorld, Agent Workflow Memory (AWM) rises above the memoryless baseline as memories accumulate, then falls below it with continued updates. On WebShop, AWM drops from 0.64 (8 examples) to 0.20 (128 examples) — exactly the memoryless baseline. Scaling memory erased its entire benefit.
Not an initialization problem: Even with GPT-5.4 generating initial memories and GPT-5-nano performing updates, decay persists. Cross-validation with Qwen3.5-{27B, 9B, 4B} solvers shows the same trend. Memory quality depends on the update mechanism, not the initial consolidator's capability.

The cleanest failure case: ARC-AGI Stream

In a controlled setting with ground-truth solutions (so every trajectory is useful):

| Condition | GPT-5.4 accuracy | |---|---| | No memory | 100% (19/19) | | Static-All (one-shot full-pool consolidation) | 94.7% | | Stream R10 (per-problem updates, 10 rounds) | 52.6% | | Stream R50 | Lower |

The difference is not the input content but the update method: the same trajectory pool, consolidated incrementally versus all at once, yields vastly different memory quality.

Three failure mechanisms

1. Misgrouping — the consolidator mixes structurally different trajectories into one abstraction. Grouping by task family (Static-Group) significantly outperforms cross-family mixing (Static-All). 2. Overgeneralization — abstractions strip away applicability conditions, so a valid lesson becomes a harmful universal rule. In task-switching experiments, a +203-point gap between "Fresh" and "Cumulative" groups came mainly from accumulated over-generalized and garbage memories. 3. Overfitting to narrow streams — with narrow input streams, memory overfits to seen instances and fails on out-of-distribution instances of the same task.

Together these show LLMs unreliably judge what to generalize, keep, or discard; each update overwrites the previous one, amplifying small errors.

Cognitive science grounding

The paper maps its findings onto complementary learning systems theory (McClelland et al., 1995): humans maintain fast episodic memory (full detail retained) alongside slow schema formation. Current agent memory systems perform schema formation with no episodic retention — analogous to forcing a brain to consolidate instantly and delete the original experience. As Bartlett (1932) showed, schema-driven retelling distorts the original narrative; LLM consolidation is the same mechanism at greater speed and scale, with no source text to check against. Metacognitive control is also absent: a consolidator evaluating its own abstraction quality is as unreliable as chain-of-thought explanations diverging from underlying computation (Turpin et al., 2023).

Engineering comparison: Auto vs Force

Agents maintain an Episodic Buffer (raw trajectories) and an Abstract Store, choosing per step to Retain, Delete, or Consolidate. Force mode mandates abstraction every round; Auto mode lets the agent decide.

| Condition | GPT-5.4 cumulative success (400 steps) | |---|---| | No memory | 22.0% | | Force (forced abstraction) | 23.8% | | Auto + Episodic=50 | 35.5% | | Auto + Episodic=100 | 37.8% |

Ablations: Abstract Only never exceeds the memoryless baseline; Episodic Only recovers nearly all of Auto's gains; Auto adds a few points. Raw trajectories carry all the useful information; abstract memories add little value on their own. In Auto mode, larger episodic buffers halve compression frequency (32.5% → 20.8%), mirroring the fast-episodic / slow-schema dual-process theory.

Engineering implications

Mainstream frameworks (CLIN, AWM, Dynamic Cheatsheet, ACE) all force memory rewriting after each interaction — the assumption "more experience = better memory" is wrong. Suggested principles:

1. Preserve raw episodes rather than treating all trajectories as compression input 2. Selective consolidation, triggered only on explicit signals 3. Traceability — abstractions must link back to source trajectories; support version rollback

For systems like OpenClaw (e.g., MEMORY.md updates after each session), the advice: keep retrieval of raw conversation records, never fully overwrite old content, preserve decision context rather than only conclusions, and periodically audit memory accuracy manually.

Limitations

Validated only on text agent benchmarks (ALFWorld, ScienceWorld, WebShop, AppWorld, Mind2Web) and ARC-AGI Stream
Only natural-language abstraction studied; parametric memory excluded
Both consolidator and solver are current-generation LLMs
Low per-problem repetition due to API costs; conclusions rely on cross-model/benchmark consistency

Conclusion

The paper does not deny memory's value — it rejects unconditional auto-updating as a safe default. The paper's closing line: "Until agents can control when and how to consolidate experience, continuously updated textual memory should be treated not as a reliable engine of self-improvement, but as a fragile mechanism that can make more experience produce worse memory."

Open questions

Implementing episodic buffers in OpenClaw's MEMORY.md update flow
Comparing abstraction quality across consolidators (GPT-5.4 vs Claude vs Qwen) on the same trajectory pool
Whether vector-database (parametric) memory follows the same decay law
Memory degradation patterns in multimodal agents
Whether fine-tuned consolidators can fix these failure modes

References

1. Paper: https://arxiv.org/abs/2605.12978 2. Frameworks: CLIN (Majumder et al., 2023), AWM (Wang et al., 2024), Dynamic Cheatsheet (Suzgun et al., 2026), ACE (Zhang et al., 2025) 3. Cognitive theory: McClelland et al. (1995); Tse et al. (2007); Bartlett (1932) 4. CoT reliability: Turpin et al. (2023)