TwinKV: A -0.004 Correlation Disrupts the Premise of KV Cache Eviction

Imagine you are a librarian. The shelves are full and some books must go, so you decide to keep books by checkout count—the most borrowed stay, the rest are discarded.

Sounds reasonable? Now someone tells you that rigorous testing shows the correlation between how often a book is borrowed and its true value to readers is -0.004—essentially unrelated, even slightly negative.

That is exactly what the TwinKV paper does in its opening: it directly tests the underlying assumption of mainstream KV cache eviction methods and finds it wrong.

KV Cache Eviction: A Survival Game for Long-Context Inference

When LLMs process long text, every token stores a pair of vectors (key and value) in each attention layer—this is the KV cache. The cache grows linearly with sequence length and, in long-context scenarios, often exceeds the size of the model's parameters themselves.

This is especially deadly for small models: they have fewer parameters and less memory, but the per-token cache size is identical to that of large models, so the cache occupies a proportionally larger share of memory.

KV cache eviction is the standard fix: during prefill, discard part of the cache and keep only a subset within a budget. The question is: which tokens to discard?

Nearly all existing methods use the same idea: look at attention distributions. Tokens that receive more attention are kept; the rest are evicted. That is the library keeping books by checkout count.

The -0.004 Spearman Correlation

TwinKV's first contribution is a leave-one-out probe:

1. Let the model process a long-context QA task normally. 2. Record the attention each context chunk receives. 3. Remove context chunks one at a time and measure how much the probability of a correct answer drops—this is the chunk's true causal contribution. 4. Compute the Spearman correlation between attention and true contribution.

Result: ρ = -0.004, p = 0.96.

Statistically, this means the two are completely unrelated. Highly attended tokens may contribute nothing to the answer; lowly attended tokens may be critical for producing it.

This finding shakes the foundation of an entire family of eviction methods: you use attention to decide which tokens to keep, but attention does not reflect token importance.

From "Importance" to "Redundancy": A Paradigm Shift

Since attention is unreliable, TwinKV switches to a completely different signal—not asking "how important is this token?" but "is this token's information backed up elsewhere?"

The core intuition is simple: if a fact appears repeatedly in a long document (verbatim, paraphrased, or in another form), dropping one copy loses no information. But if a fact appears only once, dropping it loses it forever.

TwinKV operates directly on key vectors: for each token, count how many non-adjacent tokens have nearly identical keys. If they exist, the token's information has "twins."

This signal has several properties:

• No attention needed: it bypasses attention distributions entirely, relying purely on structural similarity of key vectors.
• Training-free: zero training, plug-and-play.
• Local redundancy vs. global uniqueness: rather than measuring distance from a global mean (as prior attention-free methods do), it checks for nearby duplicate copies.

A Composable Repair: Patching, Not Replacing

TwinKV's cleverest design is that it does not replace existing eviction policies but acts as a repair pass on top of them.

Any eviction policy (attention-based, or global-reference-based) first produces its own retained set. TwinKV then does two things:

1. Identify orphans: evicted tokens whose information has no copy in the retained set—these should be rescued. 2. Identify redundant donors: retained tokens whose information has copies elsewhere in the retained set—these can be safely dropped.

TwinKV swaps orphans in and redundant donors out, keeping the original budget and scoring rules unchanged.

It is like a librarian doing a second review of a checkout-based shortlist: wait, this unpopular book is the only copy, keep it; that frequently borrowed book's content exists in three other books, safe to shelve.

Experimental Results: Modest but Real Improvements

TwinKV was combined with four recent eviction strategies on LongBench, LooGLE, RULER, and MMLU-Pro, at compression ratios of 0.3, 0.5, and 0.7.

On Qwen3-4B:

• For two strategies, TwinKV improved most configurations.
• For a third, results were roughly on par.
• For a fourth (an adaptive baseline already near the performance ceiling), help was limited.

However, when that "near-ceiling" strategy was tested with RULER on Llama-3.2-1B, TwinKV improved every configuration—because the small model's headroom (measured by the Alone score) was still large.

An interesting finding: the small model (Llama-3.2-1B) showed smaller average gains on LongBench, but a higher fraction of improved configurations. TwinKV's redundancy signal is more stable for small models—precisely the scenario where KV cache eviction matters most.

A Deeper Insight: Attention ≠ Causal Contribution

Behind the -0.004 number lies a deeper observation:

In a transformer, "being attended to" and "being needed" are two different things.

Attention tells you where the model is looking, not where the model needs to look. A token may attract high attention because it is visually salient (sentence starts, punctuation, repeated structure) yet contribute nothing to the final answer. Conversely, an inconspicuous token may be a critical link in the reasoning chain.

This echoes the "evaluation blind spot" pattern: we use the most easily measured metrics (attention, BLEU, accuracy) as the most important ones, but easy to measure ≠ causally critical.

TwinKV's solution is not to fix attention to make it more accurate, but to bypass it entirely, substituting a structural, verifiable signal (whether a duplicate exists). It is another example of solving a problem by switching levels: when a metric at the old level is unreliable, don't repair it—change levels.

Honest Assessment

TwinKV is not a silver bullet:

• It is a "composable repair," not a replacement—if the underlying policy is poor, TwinKV's ability to fix it is limited.
• On few-shot classification tasks, TwinKV does not help—redundancy across examples is itself informative there.
• The paper does not report results on very large models (70B+).

But its core contributions—the -0.004 probe experiment and the "redundancy instead of attention" paradigm shift—are important on their own: the assumption underlying mainstream KV eviction methods is wrong, and we have an alternative that does not need it.

---

Paper: TwinKV: A Composable Repair Pass for KV Cache Eviction via Pairwise Key Redundancy

Authors: Hong Chen, Yudong Zeng, Yongwei Huang, Zuhao Ouyang, Junyan Zhang, Xuming Hu (HKUST Guangzhou / Bendheim Institute of Management Science)

Published: 2026-08-27