> Paper: TokenPilot: Cache-Efficient Context Management for LLM Agents > Authors: Buqiang Xu, Zirui Xue, Dianmou Chen, et al. (Zhejiang University, UESTC, Xidian, HomologyAI) > arXiv: https://arxiv.org/abs/2606.17016 > Code: https://github.com/zjunlp/LightMem2
Key points
- Problem: Long-running agents accumulate context each turn; tool calls inject bulky environment feedback (HTML, logs, files). Worse, existing compression/eviction methods break prompt-prefix continuity, invalidating the KV cache entirely.
- Cost model: The paper uses K(C') = α·|C'_hit| + |C'_miss| with α ≪ 1, meaning cache hits are far cheaper than prefill on misses. Text pruning (LLMLingua-2, SelectiveContext), dynamic summarization (LCM), paging (MemoBrain), and OS-style abstractions (MemOS) all mutate the prefix and destroy cache reuse.
- Core insight: You cannot both compress tokens and keep the cache prefix intact — unless you design for it from the ingestion stage.
- Input messages are split into internal intent messages (task prompts, reasoning traces, tool calls, final responses — kept) and open-world environment feedback (HTML, logs, files — compressed).
- Prefix stabilization: a normalization operator replaces volatile runtime tokens with static placeholders (
<WORKDIR>,<TimeStamp>,<AGENT ID>) and relocates tool definitions to the end of the dynamic context block. Cache hit rates rise from 38.7% → 79.2% (PinchBench) and 67.2% → 83.1% (Claw-Eval). - Observation reduction: HTML slimming, terminal output truncation (600-char prefix + 400-char suffix), image downsampling (bitmap ≤100KB, SVG ≤50KB), hash-based dedup of repeated reads (cap 5), layout cleanup. Compressed content is stored in an external artifact registry (content-hash indexed) and can be dynamically restored. Key hyperparameters:
triggerMinChars = 2200,maxToolChars = 1200, global truncation threshold 50k chars. - Context chunks follow a three-state lifecycle:
active → completed → evictable(replaced by pointer stubs). - A residual-utility estimator (Qwen3.5-35B-A3B as a zero-shot verifier) runs batch evaluation every B=3 turns, returning only semantic task updates. Total verifier cost on a continuous PinchBench stream: <$0.03.
- Batch gating (B=3) avoids per-turn text mutations that would break the cache. Ablations: removing residual-utility estimation drops performance 4.7% and raises cost 40.4% (compensatory retries).
- The zero-shot estimator may misclassify residual utility in ambiguous or sparse interactions; hyperparameters (τ, B) need per-deployment tuning.
- Prefix stabilization requires backend prefix caching support; heterogeneous mixed task streams reduce prefix reuse.
- Future directions: multi-agent cross-context KV-cache topology, adaptive thresholds, distilling evicted tasks into reusable skills (
memory.autoDistill), and larger-scale industrial evaluation. - Xu, B., Xue, Z., Chen, D., et al. (2026). TokenPilot: Cache-Efficient Context Management for LLM Agents. *arXiv preprint* arXiv:2606.17016.
- Code: https://github.com/zjunlp/LightMem2
- Prior work: Fang, J., et al. (2025). LightMem: Lightweight and Efficient Memory-Augmented Generation. CoRR, abs/2510.18866.
- Related benchmarks: PinchBench, Claw-Eval
Architecture: two granularities
Global layer — Ingestion-Aware Compaction (IAC)
Local layer — Lifecycle-Aware Eviction (LAE)
Results
| Benchmark / mode | Baseline cost | TokenPilot cost | Reduction | Performance | |---|---|---|---|---| | PinchBench isolated | $8.31 (Vanilla) | $3.22 | −61.3% | 80.5 → 81.0 (best overall) | | PinchBench continuous | $7.24 | $2.79 | −61.5% | 79.2 → 81.3 (+2.1) | | Claw-Eval isolated | $5.16 | $2.27 | −56% | 64.5 → 63.1 (−1.4) | | Claw-Eval continuous | $81.52 | $10.58 | −87% | 63.4 → 60.8 |
Notably, in Claw-Eval continuous mode Vanilla cost explodes 16× due to unbounded context accumulation; TokenPilot is the only method that achieves lower cost and higher (or comparable) performance across baselines (LLMLingua-2, SelectiveContext, LCM, MemoBrain, AgentSwing, Keep-Last-N, MemOS).
Ablations (PinchBench continuous): the global IAC layer alone cuts cost 41.7% and cache miss 73.3%; adding LAE cuts cost another 33.9% and cache read 68.0%. B=3 is the optimal batch size, balancing precision, continuity, and memory overhead.
OpenClaw integration (plug-and-play)
TokenPilot ships as an OpenClaw plugin (LightMem2) with three modes:
| Mode | Stabilizer | Reduction | Eviction | Use case | |---|---|---|---|---| | conservative | on | lighter | off | maximum stability | | normal | on | balanced | off | default; single-task / short sessions | | aggressive | on | aggressive | on | long shared sessions, cost-sensitive |
Runtime commands: /lightmem2 status, report, doctor, visual, mode normal. Codex CLI and Claude Code adapters are planned.
Limitations and future work
Takeaway
TokenPilot's main contribution is a reframing of the problem: cost optimization is not about reducing token count per se, but about maximizing cache-hit continuity. Prefix stabilization, ingestion-time compaction, and conservative lifecycle-based eviction together deliver 56-87% cost reductions without sacrificing — and often improving — task performance.