Why LLMs Get Dumber the More They "Summarize": Three Abstraction Poisoning Mechanisms (UIUC + Tsinghua)
> Paper: *Useful Memories Become Faulty When Continuously Updated by LLMs* > Authors: Dylan Zhang, Yanshan Lin, Zhengkun Wu, Yihang Sun, Bingxuan Li, Dianqi Li, Hao Peng > Institutions: University of Illinois Urbana-Champaign / Tsinghua University (IIIS; work completed at UIUC) > Link: https://arxiv.org/abs/2605.12978
1. Recap: GPT-5.4 Drops from 100% to 52.6%
The previous post described the phenomenon: GPT-5.4 was placed on a 19-task ARC-AGI slice it could solve at 100%. After each solved task, the model "summarized its experience" into a memory bank. After 10 rounds, accuracy on the same tasks fell to 52.6%.
The irony: the model was given ground-truth correct answers as experience. The experience itself was useful—the problem lies in the act of "summarization" itself.
This post explains why: when LLMs compress raw episodic traces into consolidated abstractions, they make three systematic errors.
2. Mechanism 1: Misgrouping
The first step of consolidation is grouping—which experiences belong to the same "pattern" and can be merged into a general lesson. But LLMs group by semantic similarity, not structural equivalence. Two experiences may be superficially similar ("both need to find a target object") while their underlying procedures differ (one filters by color, one by shape). Merging them yields a vague "find the object" rule that is precise for neither.
Evidence: The paper compares two consolidation strategies:
- Static-All (no grouping): all task trajectories abstracted together
- Static-Group (grouped by task family): each task family abstracted separately
Example from ScienceWorld: The model merged "observe the turtle life cycle" and "observe the moth life cycle" into one memory: "Observing all life stages proceed in order is necessary for life-cycle tasks." But turtle and moth life cycles differ in stages, ordering, and criteria. Faced with a new life-cycle task (e.g., a frog), this abstraction is too vague to act on.
The essence: LLMs substitute natural-language similarity for procedural-structure equivalence.
3. Mechanism 2: Stripping Applicability Conditions
Even with correct grouping, abstraction strips the applicability conditions—when the method works, what prerequisites are needed, what boundaries must not be crossed.
The paper identifies three typical products of condition-loss in ScienceWorld:
1. Overgeneralization — Memory: "Using a lighter or fire source is necessary to change the state of outdoor substances." But state change also includes cooling and freezing; not all transitions need fire. When a task requires cooling, this memory pushes the agent the wrong way. 2. Lost conditional binding — A raw trajectory's "check thermometer first, then decide heating vs. cooling" branch becomes just "heat the target substance." 3. Goal vagueness — Memory: "Using the correct container helps boil the target substance." The trajectory specified a "glass beaker"; the abstraction blurs it into "correct container."
The essence: LLMs prefer concise, general statements, but agents need precise, conditional ones. A "general rule" without applicability boundaries is more dangerous in decision-making than a conditional specific rule—it gets misapplied to mismatched contexts.
4. Mechanism 3: Narrow Overfitting
When the input stream is narrow—successive variants of the same task type—LLM abstractions overfit to features of seen instances rather than extracting the true general pattern.
Evidence — ARC-AGI Stream: Same 19 tasks, 100% solvable without memory:
| Strategy | Update mode | Accuracy after 10 rounds | |:---|:---|:---| | Static | One-shot consolidation of all trajectories | 94.7% | | Stream | Update after each solved task | 52.6% |
Each stream update only sees the current and prior partial trajectories, so abstractions get anchored to early experience; later updates rewrite already-distorted abstractions, compounding small errors. Figure 11 further shows that with compressed, repetitive inputs, memories become overspecified and perform worse on new instances—over-learned, not under-learned.
Example: Memory: "For conductivity-ordering tasks, you must first pick up and test unknown substance B's circuit before placing it in the correct box." The rule is bound to the instance name "substance B." When the task becomes "substance C," no transfer occurs—the model memorized a specific answer, not a principle.
The essence: narrow continuous input makes LLMs mistake instance features for the pattern's definition.
5. Interaction: A Vicious Cycle
The three mechanisms amplify each other:
1. Misgrouping puts heterogeneous experiences into one abstraction pool. 2. Condition-stripping extracts an overgeneralized rule from that wrong pool. 3. Narrow input further distorts the rule with details of subsequent same-type experiences.
The result: memory "lessons" become statistical-average residue—not general enough (contaminated by specific instances), not precise enough (conditions stripped), and not transferable (wrong grouping).
6. The Fix: Raw Trajectories as First-Class Evidence
The paper also offers a treatment. In an Auto mode on ARC-AGI Stream, the agent maintains two stores—an Episodic Buffer (raw trajectories) and an Abstract Store (distilled lessons)—choosing at each step to Retain, Delete, or Consolidate.
Three key findings:
1. The episodic store carries most of the benefit. Removing raw trajectories and keeping only abstractions drops accuracy back to the memoryless baseline. 2. Episodic Management Only matches or exceeds full Auto. Only retaining/deleting raw trajectories—with zero abstraction—performs as well as or better than selective consolidation. Abstraction is not necessary. 3. Forced abstraction (Force) consistently loses to Auto. Auto's cumulative success rate over 400 training steps is higher on both GPT-5.4 and Qwen3.5-27B backbones.
Design principles: 1. Treat raw episodes as first-class evidence, not disposable raw material. 2. Make abstraction selective and delayed—not triggered after every interaction. 3. Anchor abstraction in recoverable trajectories—keep raw data for backtracking. 4. Group by task structure, not semantic similarity.
7. Comparison with Related Work
| Method | Mechanism | Problem | |:---|:---|:---| | Reflexion | Verbal self-reflection | Reflection is abstraction too—loses conditions, overgeneralizes | | ExpeL | Distills trajectories into experience entries | Flat entries lack structure and conditional boundaries | | Voyager | Auto-expanding skill library | No explicit relations between skills; misgrouping persists in composition | | A-Mem | Agentic memory management | Does not fix systematic consolidation flaws | | SKILLGRAPH (contemporary) | Graph-structured skill relations | Encodes conditions via prereq/enhance edges—a potential cure for misgrouping |
Notably, SKILLGRAPH's explicit edge types address exactly the misgrouping problem—structured dependency graphs instead of LLM semantic grouping. Both papers point to the same conclusion: agent memory needs structure, not just text.
8. Verdict: Abstraction Is a Double-Edged Sword; Agents Need Selective Forgetting
The key issue is not that "memory is useless," but that the continuous-updating process itself is toxic. LLM consolidation—the distillation of experiences into reusable lessons—is a human-level cognitive capability, but applied indiscriminately to every interaction, three systematic biases accumulate into disaster.
The deeper implication: agent memory systems should not mimic the brain's "sleep consolidation" model (merging all experiences into schemas overnight), but rather selective human memory—retain raw details, and abstract only when necessary and supported by sufficiently heterogeneous data.
This doesn't mean abstraction is unimportant. Rather, at current LLM abstraction quality, less abstraction is safer than more. Retaining raw trajectories as in-context demonstrations is already competitive—and risk-free. Wait until LLMs learn to consolidate without overwriting evidence before opening the abstraction floodgates. Until then, the "self-improving agent" narrative needs to cool down.
Reference: Zhang, D., Lin, Y., Wu, Z., Sun, Y., Li, B., Li, D., & Peng, H. (2025). Useful Memories Become Faulty When Continuously Updated by LLMs. *arXiv preprint arXiv:2605.12978*. https://arxiv.org/abs/2605.12978