Paper Information
- Title: DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression
- Authors: Anna Cordoba, Adam Puente Tercero, Nerea Angulo Hijo
- arXiv: 2607.06523
- Area: LLM inference optimization / long-context processing
- Query (Q): what the current token is "asking"
- Key (K): what "clues" other tokens offer
- Value (V): what "content" other tokens carry
- Quantization: FP16 → INT8/INT4 reduces size but introduces precision loss.
- Sparsification (e.g., H2O): keeps "heavy hitter" tokens; importance is hard to judge and task-dependent.
- Sliding Window (e.g., StreamingLLM): keeps only recent tokens; loses access to distant context such as document openings.
- Low-Rank Factorization: shrinks matrices but typically applies a uniform budget across all layers and tokens.
- 8.3× KV memory reduction (≈32 GB → ≈4 GB)
- 72.8 tokens/sec generation at 64K context—faster than many full-cache baselines since bandwidth is no longer the bottleneck
- Task quality nearly unchanged versus full caches; some tasks even improved slightly
- Better retrieval accuracy than prior compressed-cache methods
- H2O: keeps heavy hitters only; DepthWeave-KV is smarter about low-frequency tokens that may be critical proper nouns.
- StreamingLLM: sliding window discards old context; DepthWeave-KV discards nothing, only compresses.
- Quantization: orthogonal—DepthWeave-KV can stack with it for further compression.
- Keyformer: importance-score sparsification; DepthWeave-KV's token-adaptive routing is finer-grained.
- Cross-layer sharing clusters similar memories, keeping one template plus per-instance differences
- Token adaptivity labels memories by importance, placing vital ones front and center
- Online error tracking continuously verifies memory accuracy and adjusts
- Cordoba, A., Tercero, A.P., & Hijo, N.A. "DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression." arXiv:2607.06523, 2026.
- Vaswani, A., et al. "Attention is All You Need." NeurIPS, 2017.
- Zhang, Z., et al. "H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models." NeurIPS, 2023.
- Xiao, G., et al. "StreamingLLM: Efficient Streaming Language Models with Attention Sinks." ICLR, 2024.
---
Prologue: The Memory Master's Dilemma
Imagine a memory master who can hold an entire encyclopedia in mind—but every time a new chapter is read, all previous chapters must be re-memorized to understand the new one. Absurd for humans, but this is daily life for large language models. During generation, an LLM must re-read the representations of all previous tokens, stored in the KV Cache, every time it produces a new word. As context grows from 10K to 1M tokens, KV cache demand grows linearly, and eventually GPU memory runs out—the core bottleneck of long-context LLM inference.
This paper proposes an elegant solution: teaching the model to dance on the pinpoint of memory—remembering what matters, forgetting what doesn't, without tripping over its own feet.
---
Chapter 1: What Is KV Cache and Why It Matters
1.1 Self-Attention in Transformers
For each token, the model computes three vectors:
Attention scores measure Q–K matches, letting each token weigh the relevance of all others.
1.2 Why KV Cache Exists
During training, the whole sequence is processed at once. During inference, generation is token-by-token, so previously computed K and V vectors are cached to avoid recomputation. Generating token 100 means: compute its Query, fetch cached Keys/Values for tokens 1–99, compute attention, then append the new K/V to the cache.
1.3 The Long-Context Nightmare
For a model like LLaMA-3 70B (80 layers, 8 heads per layer, head dim 128) at 100,000 tokens in FP16:
KV cache size = 2 (K+V) × layers × heads × dim × seq length × 2 bytes = 2 × 80 × 8 × 128 × 100,000 × 2 bytes ≈ 32.8 GB
Since every new token requires reading the entire cache, memory bandwidth becomes the bottleneck, and generation speed collapses as context grows.
1.4 Limits of Existing Compression Methods
In reality, instruction words and retrieval-critical facts deserve more precision than transitional particles; shallow layers handling low-level features may compress more easily than deep semantic layers.
---
Chapter 2: The Core Ideas of DepthWeave-KV
2.1 Cross-Layer Residual Factorization
Adjacent Transformer layers have highly similar KV representations—like neighboring movie frames that differ only slightly. DepthWeave-KV lets groups of adjacent layers share a low-rank set of channel bases (the reference frame), while each layer stores only a lightweight residual (the difference). If four layers share a base, four full KV copies become one base plus four small residuals.
2.2 Token-Adaptive Compression
A token-conditional depth router assigns reconstruction rank per token: instruction words and retrieval keywords get higher rank (higher fidelity); ordinary tokens get lower rank; padding tokens get maximal compression. Like a smart librarian, frequently needed "books" are kept within reach.
2.3 Calibration-Free Online Error Tracking
No retraining or fine-tuning of the base model is required. During generation, attention-output probes monitor reconstruction error: if error grows, the rank increases; if error is small, compression deepens. Like driving by watching the dashboard—no need to redesign the car.
---
Chapter 3: Technical Details
3.1 The Factorization
In a standard Transformer, layer *l* has K_l, V_l ∈ R^{n×d}. DepthWeave-KV decomposes:
K_l = B_K × R_K,l + E_K,l V_l = B_V × R_V,l + E_V,l
where B_K, B_V are shared low-rank bases, R_K,l / R_V,l are per-layer residual coefficients, and E_K,l / E_V,l are (small) reconstruction errors. At inference, the model stores: one shared base per layer group, small per-layer residual coefficients, and a few original KV entries kept exactly as selected by the router.
3.2 The Routing Pipeline
1. Analyze: is the token an instruction word, retrieval-critical entity, or at a key position (sentence/paragraph start)? 2. Decide: assign high/medium/low reconstruction rank accordingly. 3. Execute: reconstruct KV from the shared base, adding residuals when needed.
3.3 Fused CUDA Implementation
Traditionally, cache reads require multiple kernel launches (base lookup, residual dequantization, attention). DepthWeave-KV fuses these into a single CUDA kernel, keeping data in GPU registers and cutting memory-bandwidth consumption—like replacing three service windows with one integrated counter.
---
Chapter 4: Experimental Results
Benchmarks: LongBench, Needle-in-a-Haystack, L-Eval, plus long-document QA and summarization.
Key findings:
Comparison with prior work:
---
Chapter 5: Deeper Reflections
Why Cross-Layer Sharing Works
Transformer layers progressively refine information: shallow layers handle local syntax, deep layers handle semantics. Adjacent layers change gradually, like moving from rough sketch to detailed painting—sharing most structure is reasonable.
The Philosophy of Token Adaptivity
Information in language is not uniformly distributed—headlines and topic sentences carry the core. DepthWeave-KV mimics how human brains instinctively distinguish critical information from fuzzy detail.
Implications for Inference Architecture
1. Tiered storage: L1 (recent KV, full precision) / L2 (moderate compression) / L3 (long-term history, heavy compression), CPU-cache style. 2. Dynamic precision: adjust KV storage precision based on task needs and resource pressure. 3. Hardware co-design: future accelerators may natively support adaptive, tiered KV storage.
---
Conclusion: Forgetting Is an Art
Humans forget breakfast grains but remember life-changing conversations. LLMs today either remember everything (prohibitive memory) or compress uniformly (losing what matters). DepthWeave-KV moves toward human-like remembering:
Like Borges' Funes, cursed by his perfect memory of every leaf's tremor, an LLM does not need perfect recall—it needs *exactly the right amount of forgetting*. DepthWeave-KV weaves cross-layer information and token importance into a graceful compression strategy. Dancing on the pinpoint of memory takes not just technique, but wisdom.
---