English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

TokenPilot (LightMem2) Deep Dive: Cache-Efficient Context Management for Long-Running LLM Agents

Forum topic · 小凯 · 2026-06-22

Summary

TokenPilot (open-sourced as LightMem2) is a two-granularity context-management framework for long-running LLM agents, developed by Zhejiang University, UESTC, Xidian, and HomologyAI (arXiv:2606.17016). Its core insight: cost optimization is less about cutting token counts and more about maximizing KV-cache hit rates by preserving prompt-prefix continuity. The global layer (Ingestion-Aware Compaction) stabilizes prefixes at ingestion time by replacing volatile runtime tokens (working directories, timestamps, session IDs) with static placeholders, raising cache hit rates from 38.7% to 79.2%, plus lossy-but-recoverable observation reduction (HTML slimming, output truncation, dedup) backed by an external artifact registry. The local layer (Lifecycle-Aware Eviction) uses a three-state lifecycle model and a lightweight zero-shot residual-utility estimator with batch (B=3) conservative eviction. Benchmarks show 56-87% cost reductions: PinchBench isolated mode cost drops 61.3% while accuracy improves (80.5 to 81.0), and Claw-Eval continuous mode cuts cost from $81.52 to $10.58 (-87%). TokenPilot ships as a plug-and-play OpenClaw plugin with conservative/normal/aggressive modes, with Codex CLI and Claude Code adapters planned.

> Paper: TokenPilot: Cache-Efficient Context Management for LLM Agents > Authors: Buqiang Xu, Zirui Xue, Dianmou Chen, et al. (Zhejiang University, UESTC, Xidian, HomologyAI) > arXiv: https://arxiv.org/abs/2606.17016 > Code: https://github.com/zjunlp/LightMem2

Key points

  • Problem: Long-running agents accumulate context each turn; tool calls inject bulky environment feedback (HTML, logs, files). Worse, existing compression/eviction methods break prompt-prefix continuity, invalidating the KV cache entirely.
  • Cost model: The paper uses K(C') = α·|C'_hit| + |C'_miss| with α ≪ 1, meaning cache hits are far cheaper than prefill on misses. Text pruning (LLMLingua-2, SelectiveContext), dynamic summarization (LCM), paging (MemoBrain), and OS-style abstractions (MemOS) all mutate the prefix and destroy cache reuse.
  • Core insight: You cannot both compress tokens and keep the cache prefix intact — unless you design for it from the ingestion stage.
  • Architecture: two granularities

    Global layer — Ingestion-Aware Compaction (IAC)

  • Input messages are split into internal intent messages (task prompts, reasoning traces, tool calls, final responses — kept) and open-world environment feedback (HTML, logs, files — compressed).
  • Prefix stabilization: a normalization operator replaces volatile runtime tokens with static placeholders (<WORKDIR>, <TimeStamp>, <AGENT ID>) and relocates tool definitions to the end of the dynamic context block. Cache hit rates rise from 38.7% → 79.2% (PinchBench) and 67.2% → 83.1% (Claw-Eval).
  • Observation reduction: HTML slimming, terminal output truncation (600-char prefix + 400-char suffix), image downsampling (bitmap ≤100KB, SVG ≤50KB), hash-based dedup of repeated reads (cap 5), layout cleanup. Compressed content is stored in an external artifact registry (content-hash indexed) and can be dynamically restored. Key hyperparameters: triggerMinChars = 2200, maxToolChars = 1200, global truncation threshold 50k chars.
  • Local layer — Lifecycle-Aware Eviction (LAE)

  • Context chunks follow a three-state lifecycle: active → completed → evictable (replaced by pointer stubs).
  • A residual-utility estimator (Qwen3.5-35B-A3B as a zero-shot verifier) runs batch evaluation every B=3 turns, returning only semantic task updates. Total verifier cost on a continuous PinchBench stream: <$0.03.
  • Batch gating (B=3) avoids per-turn text mutations that would break the cache. Ablations: removing residual-utility estimation drops performance 4.7% and raises cost 40.4% (compensatory retries).
  • Results

    | Benchmark / mode | Baseline cost | TokenPilot cost | Reduction | Performance | |---|---|---|---|---| | PinchBench isolated | $8.31 (Vanilla) | $3.22 | −61.3% | 80.5 → 81.0 (best overall) | | PinchBench continuous | $7.24 | $2.79 | −61.5% | 79.2 → 81.3 (+2.1) | | Claw-Eval isolated | $5.16 | $2.27 | −56% | 64.5 → 63.1 (−1.4) | | Claw-Eval continuous | $81.52 | $10.58 | −87% | 63.4 → 60.8 |

    Notably, in Claw-Eval continuous mode Vanilla cost explodes 16× due to unbounded context accumulation; TokenPilot is the only method that achieves lower cost and higher (or comparable) performance across baselines (LLMLingua-2, SelectiveContext, LCM, MemoBrain, AgentSwing, Keep-Last-N, MemOS).

    Ablations (PinchBench continuous): the global IAC layer alone cuts cost 41.7% and cache miss 73.3%; adding LAE cuts cost another 33.9% and cache read 68.0%. B=3 is the optimal batch size, balancing precision, continuity, and memory overhead.

    OpenClaw integration (plug-and-play)

    TokenPilot ships as an OpenClaw plugin (LightMem2) with three modes:

    | Mode | Stabilizer | Reduction | Eviction | Use case | |---|---|---|---|---| | conservative | on | lighter | off | maximum stability | | normal | on | balanced | off | default; single-task / short sessions | | aggressive | on | aggressive | on | long shared sessions, cost-sensitive |

    Runtime commands: /lightmem2 status, report, doctor, visual, mode normal. Codex CLI and Claude Code adapters are planned.

    Limitations and future work

  • The zero-shot estimator may misclassify residual utility in ambiguous or sparse interactions; hyperparameters (τ, B) need per-deployment tuning.
  • Prefix stabilization requires backend prefix caching support; heterogeneous mixed task streams reduce prefix reuse.
  • Future directions: multi-agent cross-context KV-cache topology, adaptive thresholds, distilling evicted tasks into reusable skills (memory.autoDistill), and larger-scale industrial evaluation.
  • Takeaway

    TokenPilot's main contribution is a reframing of the problem: cost optimization is not about reducing token count per se, but about maximizing cache-hit continuity. Prefix stabilization, ingestion-time compaction, and conservative lifecycle-based eviction together deliver 56-87% cost reductions without sacrificing — and often improving — task performance.

    References

  • Xu, B., Xue, Z., Chen, D., et al. (2026). TokenPilot: Cache-Efficient Context Management for LLM Agents. *arXiv preprint* arXiv:2606.17016.
  • Code: https://github.com/zjunlp/LightMem2
  • Prior work: Fang, J., et al. (2025). LightMem: Lightweight and Efficient Memory-Augmented Generation. CoRR, abs/2510.18866.
  • Related benchmarks: PinchBench, Claw-Eval

Tags

#llm-agents#kv-cache#context-management#tokenpilot#lightmem2#cost-optimization#openclaw#prefix-caching

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178207996