Overview
LightMem (ICLR 2026, arXiv:2510.18866, code: https://github.com/zjunlp/LightMem) is a memory-augmented generation framework for LLM agents, authored by Jizhan Fang and collaborators at Zhejiang University, Nanjing University, and the National University of Singapore. It borrows the Atkinson-Shiffrin human memory model and restructures agent memory as a three-stage pipeline with offline consolidation instead of real-time updates.
Why current memory systems struggle
Existing systems such as Mem0, A-MEM, MemoryOS, and LangMem maintain memory synchronously during conversation. Every user turn triggers compression, summarization, indexing, or storage, which inflates latency, API calls, and token cost. As Dex Horthy notes in *12-Factor Agents*, context windows above ~40% capacity degrade agent quality; real-time memory maintenance makes that worse by stuffing offline work into every online second.
Three-stage architecture
Stage 1: Sensory Memory (online)
A bounded buffer receives raw turns and performs only two cheap operations:
1. Compression with LLMLingua-2. A sub-2GB BERT-class prompt compressor removes redundant tokens. Compression ratio is tunable between 0.4 and 0.8; ~0.6 is reported as the empirical sweet spot. 2. Topic segmentation via attention peaks. Instead of fixed sentence-count chunking, LightMem monitors LLMLingua-2's internal attention matrix. Sentences whose attention to prior sentences drops sharply mark new topic boundaries. A hybrid layer adds semantic-similarity checks. Hybrid segmentation exceeds 80% accuracy versus 62% for either signal alone.
The buffer drops irrelevant content and groups related turns. All work runs on small local models, so user-perceived latency stays near zero.
Stage 2: Short-Term Memory (online, lightly triggered)
Topic-grouped turns enter per-topic buffers sized 256/512/768/1024 tokens. When a buffer fills, a lightweight summary is produced by the backbone LLM and embedded with all-MiniLM-L6-v2. These candidate entries wait for long-term storage rather than being written immediately.
Stage 3: Long-Term Memory (offline "sleep")
Long-term memory is not updated during conversation. After the session ends, LightMem runs deduplication, merging, abstraction, and linking in parallel queues. Updates are time-ordered so newer entries overwrite older ones rather than being overwritten by stale data. This is where the backbone LLM is invoked, but in batch.
Headline numbers (GPT-4o-mini, LongMemEval-S, r=0.7, th=512)
| Metric | A-MEM (strongest baseline) | LightMem | Factor | |---|---|---|---| | Accuracy | 62.60% | 68.64% | +6.04% | | Total tokens | 1.606M | 28K | 38x reduction | | API calls | 986 | 18 | 30x reduction | | Runtime | 5132s | 284s | 12x speedup |
Online-only costs (excluding offline sleep):
| Metric | A-MEM | LightMem | Factor | |---|---|---|---| | Tokens | 1.606M | 15K | 106x reduction | | API calls | 986 | 6 | 159x reduction |
On Qwen3-30B-A3B-Instruct-2507, online tokens drop 117x and API calls drop 310x, because baselines like MemoryOS and A-MEM repeatedly trigger full updates across multi-turn sessions while LightMem collapses hundreds of calls into a few offline batches.
Trade-offs
LightMem is not free. On LoCoMo's Single-Assistant category, which probes assistant behavioral consistency (e.g., "you promised to do X"), LightMem scores 32.14% versus A-MEM's 96.43%. Compression and topic splitting discard fine-grained behavioral signals needed for such questions. LLMLingua-2 can also collapse sentences to empty strings, forcing fallback to the original text. Topic segmentation sometimes over-segments, fragmenting coherent passages and hurting summary quality. The system fits high-density, cross-topic scenarios (customer support, consulting, long-running assistants) better than behavior-sensitive ones (role-play, commitment tracking).
Experimental results
- LongMemEval-S (500 questions, ~115K tokens each): LightMem beats A-MEM by 2.09–6.40 points on GPT-4o-mini and up to 7.67 points on Qwen3. Strongest gains on Temporal Reasoning (+19.8%), Multi-Session (+22.9%), and Knowledge Update (+19.0%).
- LoCoMo (multi-turn long conversation): GPT accuracy gains of 6.10–18.12 points, with 2.87–20.92x token efficiency and 13.29–39.78x fewer API calls.
- Baselines: Mem0 weakest (36–53% accuracy), LangMem and MemoryOS mid-tier, A-MEM strongest. LightMem wins on nearly every metric except Single-Assistant.
- Own Your Context Window: online context stays minimal — sensory buffer 512 tokens, short-term cap 1024.
- Unify Execution State and Business State: the short-term buffer is both temporary chat storage and business-state aggregator, avoiding parallel state machines.
- Small, Focused Agents: compression, segmentation, summarization, embedding, and indexing each use independent small models. LLMLingua-2 is under 2GB, embeddings are shared, and the backbone LLM is only called in batch offline.
- Pre-fetch Context: the offline sleep phase is essentially pre-fetching — building indexed, linked context for fast retrieval at query time.
- LightMem: Lightweight and Efficient Memory-Augmented Generation. ICLR 2026. arXiv: https://arxiv.org/abs/2510.18866
- GitHub: https://github.com/zjunlp/LightMem
- Atkinson & Shiffrin (1968), Human Memory Model
- LLMLingua-2, Jiang et al., ACL 2024
- LongMemEval, Wu et al., 2025
- 12-Factor Agents, Dex Horthy / HumanLayer
Alignment with 12-Factor Agents
Takeaway
The contribution is architectural, not a single trick: convert agent memory from real-time, reactive maintenance to offline, batched consolidation. Trade user-waiting time for server-idle compute, trade per-call overhead for amortized batching. Code is open-source, and the same Zhejiang NLP group has related work *StructMem* accepted at ACL 2026.