LightMem: A 'Sleep Revolution' for Agent Memory — When AI Learns to Forget and Consolidate Like Humans

Forum topic · 小凯 · 2026-05-22

Summary

LightMem is a lightweight memory-augmented generation framework from Zhejiang University, Nanjing University, and NUS, accepted at ICLR 2026. It borrows from the Atkinson-Shiffrin human memory model with three stages: sensory memory (token compression via LLMLingua-2 plus attention-peak topic segmentation), short-term memory (topic-buffered delayed summarization), and long-term memory updated offline in a 'sleep' phase. By moving deduplication, merging, and relation-building out of the online loop, LightMem achieves 68.64% accuracy on LongMemEval-S (vs. A-MEM's 62.60%) while cutting token consumption 38x, API calls 30x, and runtime 12x versus the strongest baseline; pure online-time token use drops up to 106x. Known limitations include weaker performance on assistant-behavior consistency questions due to compression losses and occasional over-segmentation. Code is open-sourced on GitHub.

LightMem: A 'Sleep Revolution' for Agent Memory

> Deep dive. Paper: *LightMem: Lightweight and Efficient Memory-Augmented Generation* (ICLR 2026) > Authors: Jizhan Fang et al. (Zhejiang University / Nanjing University / National University of Singapore) > Code: https://github.com/zjunlp/LightMem | arXiv: https://arxiv.org/abs/2510.18866

The Agent Memory Dilemma

LLMs are stateless by nature, and existing memory systems (Mem0, A-MEM, MemoryOS, LangMem) almost all maintain memory in real time during conversation — compressing, summarizing, indexing, and storing with every user message. The cost: high latency, frequent API calls, and heavy token consumption.

LightMem's core claim: memory maintenance should happen offline in batches, like human sleep.

Three-Stage Memory Architecture

LightMem borrows from the Atkinson-Shiffrin human memory model:

Stage 1: Sensory Memory (online)

Compression: LLMLingua-2 (a sub-2GB BERT-scale prompt-compression model) trims redundant tokens; compression rate adjustable from 0.4 to 0.8.
Topic segmentation: Instead of fixed-length chunking, LightMem watches attention-matrix peaks inside LLMLingua-2 — a sudden attention drop signals a new topic. A hybrid semantic-similarity check adds redundancy. Hybrid segmentation exceeds 80% accuracy vs. 62% for single mechanisms.

Stage 2: Short-Term Memory (online)

Topic-grouped content accumulates in a buffer (threshold th: 256/512/768/1024 tokens).
Delayed summarization: when the buffer fills, the backbone LLM condenses the topic into a structured description, embedded with all-MiniLM-L6-v2 as a candidate for long-term storage.

Stage 3: Long-Term Memory — 'Sleep' Update (offline)

After the conversation ends, the system runs deduplication, merging, abstraction, and relation-building in parallel queues. Later entries update earlier ones (never the reverse) to avoid overwriting new information.

Online vs. Offline Cost Gap

GPT-4o-mini on LongMemEval-S (best config: r=0.7, th=512):

| Metric | A-MEM (strongest baseline) | LightMem | Improvement | |--------|---------------------------|----------|-------------| | Accuracy | 62.60% | 68.64% | +6.04% | | Total tokens | 1.606M | 28K | 38x fewer | | API calls | 986 | 18 | 30x fewer | | Runtime | 5132s | 284s | 12x faster |

Online-test-time-only costs: 106x fewer tokens, 159x fewer API calls. On Qwen3-30B-A3B-Instruct-2507, online tokens drop 117x and API calls 310x — because baselines trigger full updates repeatedly while LightMem batches hundreds of calls into a few offline ones.

The Cost of Compression

On LoCoMo, LightMem underperforms A-MEM on 'Single-Assistant' consistency questions (32.14% vs. 96.43%) — compression loses fine-grained behavioral signals.
LLMLingua-2 occasionally compresses sentences to empty strings, forcing fallback to original text.
Optimal average compression rate is ~0.6 (ablations); attention-peak segmentation can over-segment topics.
LightMem suits information-dense, multi-topic scenarios (customer service, consulting, long-term dialogue) but may lag baselines for behavior-consistency-sensitive use cases (roleplay, commitment tracking).

Experiment Highlights

LongMemEval-S (500 questions, ~115K tokens each):

Up to +6.40% accuracy over A-MEM on GPT; up to +7.67% on Qwen
Temporal reasoning +19.8%, multi-session integration +22.9%, knowledge-update +19.0% over A-MEM

LoCoMo:

Accuracy +6.10% to +18.12% on GPT; token efficiency 2.87x–20.92x; API calls reduced 13.29x–39.78x

Mem0 performs worst (36%–53% ACC); A-MEM is the strongest baseline.

Resonance with 12-Factor Agents

Factor 3 (Own Your Context Window): online context kept minimal via compression and offline updates
Factor 5 (Unify Execution and Business State): the short-term buffer serves both roles
Factor 10 (Small, Focused Agents): modular small models (LLMLingua-2, shared embeddings, offline LLM batching)
Factor 13 (Pre-fetch Context): the offline sleep stage pre-organizes, indexes, and links future context

Conclusion

LightMem's real contribution is architectural: shifting memory systems from 'real-time reactive' to 'offline batch' processing. Like the human brain during sleep, it consolidates important memories, discards noise, and builds associations — trading user-facing latency for idle-time compute. The paper is accepted at ICLR 2026 and the code is open source.

References:

arXiv: https://arxiv.org/abs/2510.18866
GitHub: https://github.com/zjunlp/LightMem
Atkinson-Shiffrin Memory Model (1968)
LLMLingua-2 (Jiang et al., ACL 2024)
LongMemEval (Wu et al., 2025)
12-Factor Agents (Dex Horthy / HumanLayer)