TokenPilot / LightMem2: Cache-Efficient Long-Horizon Context Management for LLM Agents

Key points

• Problem reframing: Long-horizon LLM agents suffer from "context inflation" and KV-cache invalidation. Existing compression/eviction methods (LLMLingua-2, SelectiveContext, LCM, Pichay, MemoBrain, MemOS) mutate prompt structure and break byte-level prefix reuse, so cache hit rates collapse and effective costs remain high.
• Cost model: K(C') = α·|C'_hit| + |C'_miss|, with α ≪ 1. Maximizing cache-hit continuity is more important than minimizing raw token count.
• Global layer — Ingestion-Aware Compaction (IAC):
• Splits messages into internal intent Ω_int (tasks, thoughts, tool calls, final responses) vs. open-world environment feedback Ω_env (HTML, logs, files).
• Prefix stabilization replaces volatile runtime tokens with static placeholders such as <WORKDIR>, <TimeStamp>, <AGENT ID> and relocates tool definitions to a dynamic context block, raising cache hit rate from 38.7% → 79.2% on PinchBench and 67.2% → 83.1% on Claw-Eval.
• Observation reduction applies HTML slimming, prefix+suffix output truncation (600+400 chars), bitmap ≤100KB / SVG ≤50KB downsampling, hash-based dedup (≤5 repeats), and whitespace/code-fence cleanup, with full content kept in an external artifact registry keyed by content hash. Trigger threshold ~2,200 chars, tool-routing threshold 1,200 chars, global truncation cap 50k chars.
• Local layer — Lifecycle-Aware Eviction (LAE):
• Three-state chunk model: active → completed → evictable, with eviction triggered only when residual utility reaches zero.
• A zero-shot Qwen3.5-35B-A3B verifier runs in batched mode every B=3 turns, emitting only semantic delta taskUpdates rather than full registry dumps; reported continuous-mode operating cost stays under $0.03.
• Conservative eviction preserves document-structure knowledge so later turns can locate targets without re-reading files; removing the residual-utility estimator drops overall performance by 4.7% and raises cost by 40.4%.
• Experimental results:
• PinchBench Isolated: Overall 81.0 (vs. 80.5 Vanilla), cost $3.22 vs. $8.31 (–61.3%), cache-miss tokens 1.933M vs. 8.753M (–77.9%). TokenPilot is the only method that simultaneously lowers cost and raises overall score.
• PinchBench Continuous: Overall 81.3 (vs. 79.2), cost $2.79 vs. $7.24 (–61.5%).
• Claw-Eval Isolated: Overall 63.1 vs. 64.5, cost $2.27 vs. $5.16 (–56%).
• Claw-Eval Continuous: cost $10.58 vs. $81.52 Vanilla (–87%) and $24.12 MemOS, while keeping Overall 60.8 vs. 63.4.
• Ablation on PinchBench Continuous: +IAC cuts cost 41.7% and cache-miss tokens 73.3%; +LAE further cuts cost 33.9% and cache-read tokens 68.0%.
• Batch size B=3 is the validated sweet spot; B=1 truncates too aggressively, while B=5–7 causes memory bloat and latency growth.
• OpenClaw / LightMem2 integration: Ships as a drop-in plugin with conservative / normal / aggressive modes (Stabilizer always on, Reduction intensity varies, Eviction only in aggressive), runtime commands /lightmem2 status|report|doctor|visual|mode …, and JSON config exposing proxyBaseUrl, proxyApiKey, and per-module toggles. Planned adapters include Codex CLI and Claude Code.
• Limitations and outlook: The Qwen3.5-35B-A3B zero-shot verifier can misclassify residual utility in highly ambiguous or sparse interactions; thresholds τ and B may need retuning per deployment; prefix stabilization requires backend prefix-cache support; continuous-mode gains assume same-category task grouping. Future directions include multi-agent cross-context KV-cache communication, adaptive thresholds, distilling evicted tasks into reusable skills (similar to SkillClaw), and larger-scale industrial evaluation beyond the current 123/161-task test sets.
• Resources:
• Paper: https://arxiv.org/abs/2606.17016
• Code: https://github.com/zjunlp/LightMem2
• Predecessor: LightMem (arXiv:2510.18866)
• Benchmarks: PinchBench, Claw-Eval