Key Points
- Core problem: Scalable memory does not imply scalable computation. SSM Fast Weights can store information indefinitely, but a single forward pass is insufficient to convert observed tokens into useful weight memory, so deep-reasoning performance collapses even when the full context fits.
- Proposed method: LLM Sleep triggers an offline phase when the KV cache fills. During sleep the model runs N recursive forward passes on the same cached chunk, updating Gated Delta Network fast weights via a Hebbian-style rule, then evicts the KV cache and resumes online inference.
- Architectural details: Fast weights are a fixed d x d matrix updated as S_t = alpha_t * S_{t-1} + beta_t * v_t k_t^T, with data-dependent forget and input gates plus delta-rule corrections. Online prediction uses a single forward pass per token, keeping latency constant; only the offline sleep loop adds compute.
- Biological inspiration: The wake/sleep pattern mirrors hippocampal-neocortical consolidation. Hippocampal replay during slow-wave sleep transfers episodic traces into neocortical synaptic weights; LLM Sleep does the same for KV cache to SSM fast weights.
- Threshold effect of N: Increasing N does not yield linear gains but unlocks qualitatively new reasoning depths. On Depo multi-hop retrieval, each doubling of N roughly doubles the maximum hop count the model can solve.
- Empirical results:
- Rule 110 cellular automata: with L=24 hard-eviction windows, N=1 stays near random (~10%) at t=32, while N=3-4 exceeds 30%.
- Depo retrieval on shuffled directed graphs up to 75 nodes: N=1 fails beyond 4 hops; N=4 enables 8-16 hop reasoning.
- GSM-Infinite: Jet-Nemotron 2B gains +9 to +11 points at 6-8 operations with N=6; Ouro 1.4B with Jet layers jumps from 41.9% to 61.5% (+47%) at 6 ops with N=4, and 2-operation arithmetic climbs from 59.6% to 90.5%.
- Trade-offs and cost: Throughput scales inversely with N (about 10k tok/s at N=2, ~5k tok/s at N=4). Cross-window serial dependence is mitigated by modern GPU parallelism for large windows.
- Comparison with related work:
- Context compression and Cartridges keep KV-cache-based attention; LLM Sleep migrates to weight memory.
- Context distillation uses predefined losses; LLM Sleep uses a learned recursive forward rule.
- Test-time training applies single-step gradients per chunk; LLM Sleep applies multi-step learned updates.
- LoRA adapters update weights once per chunk; LLM Sleep iterates.
- Deep recurrent models add latency during prediction; LLM Sleep keeps online latency constant.
- Ring/Striped attention target memory efficiency, not computational depth.
- Implications: Memory consolidation is itself a non-trivial computation requiring iteration; some reasoning tasks have intrinsic serial structure that parallel computation cannot shortcut; threshold effects suggest substantial headroom if training instabilities in deep recurrence are resolved.
Reference
Lee, S., McLeish, S., Goldstein, T., & Fanti, G. (2026). *Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference*. arXiv:2605.26099.