Summary
A new paper (arXiv 2610.20582) introduces Latent Compaction, a method that compresses key-value (KV) caches in long-context language models by learning a compact, document-specific cache via test-time training. Cache entries are initialized using reconstruction importance, teacher attention, and key-space diversity. Training first fits attention outputs and log-normalizers to those of the full cache while holding reconstruction-selected entries fixed, then jointly optimizes the entire cache through end-to-end distillation. Model weights stay frozen, and training uses only generic document prompts without access to downstream questions or answers. Experiments on RULER, LongBench, and Phonebook Lookup show the approach significantly outperforms equal-budget compression baselines: on RULER it reaches 96.7% accuracy at 32x compression versus 40.0% for KVZip+. The method also transfers across model scales and families, delivering retrieval gains on Qwen3 and MiniCPM5. The results suggest treating KV caches as neural networks optimized with test-time training can preserve retrieval accuracy under aggressive compression. Code is open-sourced at github.com/THUNLP/latent-compaction.
Paper Overview
Field: NLP
Authors: Yingfa Chen, Xingyu Shen, Chaojun Xiao, Xu Han
Published: 2026-10-14
arXiv: 2610.20582
Summary
Long contexts enlarge key-value (KV) caches, increasing memory use and attention costs during generation. Compressing these caches must preserve information for unseen questions. The paper introduces Latent Compaction, which learns a compact document-specific cache through test-time training.
- Initialization: cache entries are initialized using reconstruction importance, teacher attention, and key-space diversity.
- Training: first fits attention outputs and log-normalizers to those of the full cache, with reconstruction-selected entries held fixed; then jointly optimizes the entire cache through end-to-end distillation from the full-cache model.
- Constraints: model weights remain frozen throughout, and training uses only generic prompts about the document, without access to downstream questions or answers.
Results
Experiments on RULER, LongBench, and Phonebook Lookup show the method significantly outperforms equal-budget compression baselines:
- On RULER, it reaches 96.7% accuracy at 32x compression, compared to only 40.0% for KVZip+.
- The method transfers across model scales and families, achieving retrieval gains on both Qwen3 and MiniCPM5.
These results show that treating KV caches as neural networks optimized via test-time training can preserve retrieval accuracy under aggressive compression.
Code
Open source: https://www.github.com/THUNLP/latent-compaction
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178636045