论文信息
DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
- Authors: Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo
- arXiv: 2607.06523
- Area: LLM inference optimization / long-context processing
- Cross-layer residual factorization: Adjacent Transformer layers have highly similar KV representations. Instead of storing full KV per layer, a group of neighboring layers shares one set of low-rank channel bases; each layer stores only a lightweight residual:
- Token-adaptive routing: A token-conditional depth router classifies each token (instruction words, retrieval-critical entities/numbers, filler words) and assigns a higher reconstruction rank to important tokens and a lower rank to unimportant ones, spending memory only where precision matters.
- Calibration-free online error tracking: No fine-tuning of the base model is needed. During generation, reconstruction error is monitored through attention-output probes; ranks are increased when error is high and decreased when error is low — analogous to adjusting the throttle by watching the dashboard rather than redesigning the car.
- Fused CUDA implementation: Base lookup, residual dequantization, and attention computation are fused into a single CUDA kernel, avoiding multiple kernel launches and reducing data movement through GPU memory.
- 8.3x KV memory reduction — a 32 GB cache fits in roughly 4 GB.
- 72.8 tokens/second generation at 64K context, faster than many full-cache baselines since bandwidth is no longer the bottleneck.
- Task quality essentially matches the uncompressed full cache; retrieval accuracy beats prior compressed-cache methods.
- Compared with H2O (which may drop critical low-frequency tokens), StreamingLLM (which discards old context entirely), quantization methods like GQAQ (orthogonal and stackable), and importance-score sparsification like Keyformer, DepthWeave-KV's token-adaptive strategy is finer-grained and never discards tokens outright.
- Cordoba, A., Tercero, A.P., & Hijo, N.A. "DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression." arXiv:2607.06523, 2026.
- Vaswani, A., et al. "Attention is All You Need." NeurIPS, 2017.
- Zhang, Z., et al. "H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models." NeurIPS, 2023.
- Xiao, G., et al. "StreamingLLM: Efficient Streaming Language Models with Attention Sinks." ICLR, 2024.
Background: Why the KV Cache is the bottleneck
During autoregressive generation, an LLM must re-read stored Key/Value representations of all previous tokens at every step. These are kept in the KV cache, which grows linearly with sequence length. For a model like LLaMA-3 70B (80 layers, 8 heads, 128-dim heads) with a 100,000-token context in FP16, the cache alone is roughly 32.8 GB — potentially more than the model weights — and reading it every step makes memory bandwidth a hard bottleneck.
Existing compression approaches (quantization, sparsification such as H2O, sliding windows like StreamingLLM, low-rank factorization) share a common limitation: a uniform budget. They apply the same compression strategy to all layers and all tokens, ignoring that shallow/deep layers and instruction vs. filler tokens differ greatly in importance.
Key ideas
K_l = B_K × R_K,l + E_K,l V_l = B_V × R_V,l + E_V,l
where B are shared low-rank bases, R are per-layer residual coefficients, and E is a small reconstruction error.
Results
Benchmarks: LongBench, Needle-in-a-Haystack, L-Eval, and long-context QA/summarization.
Why cross-layer sharing works
Transformer layers progressively refine representations: shallow layers capture low-level features, deep layers capture high-level semantics. Adjacent layers change information gradually rather than discontinuously — like refining a sketch into a detailed painting — so sharing a common structure across neighbors is well founded.
Implications for inference architecture
1. Tiered storage: future inference systems may adopt CPU-cache-like hierarchies (L1 full-precision recent KV, L2 lightly compressed, L3 highly compressed long-term history). 2. Dynamic precision: storage precision can flex with task needs and resource pressure. 3. Hardware co-design: AI accelerators could natively support tiered, adaptive KV storage.
Conclusion
The post closes with the metaphor of Borges' "Funes the Memorious": perfect memory is a curse. LLMs do not need to remember everything — they need to *forget judiciously*. DepthWeave-KV turns forgetting into a craft: weaving cross-layer information into shared bases and weaving token importance into the compression strategy, so the model can dance on the needle's tip of memory without losing its footing.