Attic-KV: Self-Quizzing Beats Rereading for KV Cache Compression at Tight Budgets

The Counterintuitive Failure

Imagine preparing for an exam with a 500-page textbook, but you're only allowed to bring 15 pages into the exam room (3% retention). Intuition says: reread the whole book, mark the important parts, pick the 15 most-used pages. But what if your 15 pages don't contain the answers to the actual questions?

This is what's happening in KV cache compression. A paper from October 2026, Attic-KV (arXiv: 2610.12133), reveals that under tight budgets, letting the model reread the entire context to score KV entries performs worse than doing nothing at all. The authors name this phenomenon: "rehearse everything, remember nothing."

Background: Caches Built Before Questions Arrive

During LLM inference, Keys and Values for every token are cached to avoid recomputation, but long contexts make the cache explode—a 100k-token document can take several GB. The fix is compression: keep only a small fraction of KV entries.

The catch: in many scenarios the cache is built *before* the question arrives:

• RAG systems: per-document caches precomputed, reused for any query
• Serving systems: long system prompts shared across users
• Conversational assistants: caches must serve future turns that haven't happened yet

This "compress first, ask later" pattern means compression must be query-agnostic and one-shot. And since cache volume grows with documents, only extreme compression pays off—5% retention yields 20x more cached documents, while 30% yields only 3x.

Mainstream Method: Rehearsal Scoring

The current mainstream approach is rehearsal scoring, introduced by KVzip: have the model "rehearse" the context—reread the full document while watching which KV entries attention visits most, then keep those.

The assumption: *the cache remembers what it rehearsed*. More complete rehearsal means better memory.

But Attic-KV's authors found this assumption backfires at tight budgets.

The Core Counterintuitive Data

On the RULER benchmark, Qwen3-8B, 3% retention:

• Full-context rehearsal (KVzip's approach): 31.5 / 96.5
• No rehearsal at all: on some tasks, outperforms full rehearsal

Why? When the model rereads the entire context, attention spreads evenly across every passage—including fluent but never-queried transition text. Key facts that would actually be asked about get the same budget as filler.

The authors use a precise analogy: it's like a student rereading the textbook before the exam. Every word has been seen, but the specific knowledge point the exam tests gets the same attention as the table of contents.

More concretely: at 5% retention, a cache rehearsing only answers is nearly lossless (93.5 vs 94.7 uncompressed), while a cache rehearsing the entire context scores only 44.9. Capacity isn't the bottleneck—allocation is.

The Breakthrough: Let the Student Self-Quiz

The paper's core insight comes from a classic finding in educational psychology. Dunlosky et al.'s 2013 large-scale review of learning techniques ranked:

1. Reading through the textbook (low utility) 2. Rereading and highlighting (low utility) 3. Self-testing (high utility)

Roediger & Karpicke (2006) further showed: practicing producing answers beats seeing the material again. Attention truly lands on answers only when producing them.

Attic-KV brings this to KV caches: instead of rereading the context, have the model write its own questions and answer them—generating QA pairs whose answers quote the source context verbatim. Attention lands precisely on "facts that will be asked about" rather than dispersing across "fluent but unqueried passages."

Two Principles

The authors distill two design principles:

Principle 1: Rehearse what will be read. The cache should rehearse what future queries will read, not the whole context. Rehearsing only askable facts already solves full-rehearsal failures on fact-sparse tasks.

Principle 2: Rehearse as much as there is. Rehearsal volume should track the amount of readable content in the context, not a fixed ratio. At equal average rehearsal volume, content-adaptive rehearsal (78.0) beats fixed-ratio (74.6).

Under these principles, prior methods become special cases: KVzip rehearses 100% of the context, KV2 rehearses a fixed ratio ρ, and content-adaptive rehearsal approaches KVzip when the budget is generous.

Attic's Three Components

Attic's name comes from Sherlock Holmes's "brain attic" metaphor: a skilled workman packs his attic only with tools that might be useful. Attic's rehearsal consists of:

1. Self-generated QA rehearsal: the model writes 6 plausible questions and answers them by quoting the context. Attention lands precisely on key facts when producing answers. 2. Anchor tokens (from Park et al. 2025): deciding what else needs rehearsal. 3. Content-adaptive rehearsal volume: sentence lengths containing anchors determine how much to rehearse.

Key property: Attic is training-free. It only changes rehearsal content, not the scoring mechanism—so it plugs into any rehearsal-scoring method.

Results: The Tighter the Budget, the Bigger the Advantage

On RULER (4 models) and LongBench natural-text tasks (2 models):

• Attic alone: best training-free method in all 8 test settings
• Plugged into KVgrad (gradient attribution): up to +17.1 points
• Plugged into RestoreKV+ (training-based): up to +28.1 points
• At 3% retention: +41.9 points over full rereading
• LooGLE benchmark (one compressed cache per document answers all questions): Attic leads on both short- and long-range dependencies

Most striking: at 5% retention, answer-only rehearsal scores 93.5 (vs 94.7 uncompressed), while full-context rehearsal scores only 44.9. **Same budget; the only difference is *what* gets rehearsed—a 93.5 vs 44.9 gap.

Why This Matters

The paper's significance goes beyond a better compression method. It reveals a deeper principle:

Memory is a function of retrieval, not storage. A cache's value lies not in how much context it stores, but in what future queries can read from it. Full rehearsal optimizes coverage—every token gets seen; self-quiz rehearsal optimizes query hit-rate—askable facts get precisely retained.

This mirrors human memory: rereading a book retains nothing, but reading with questions in mind retains the answers. Active retrieval beats passive exposure—a classic educational psychology finding, now validated in Transformer KV caches.

The practical value: Attic is training-free, changes only the rehearsal strategy, requires no weight modification, and adds no training cost. Systems already running KVzip or KVgrad can add Attic at near-zero cost—with up to 28 points of improvement.

One Small Caveat

The paper doesn't address: how to guarantee the quality of self-generated QAs? If the model's generated questions diverge from the real query distribution, Attic's advantage shrinks. The paper uses a fixed 6-question prompt template, which may not be optimal for all scenarios. Making rehearsal questions adaptive to the actual query distribution (e.g., using historical queries as reference) could work better.

But the contribution stands. The paper solves a structural problem hidden by intuition in the simplest way—changing the rehearsal content. As Holmes put it: "Do not fancy that the little room has elastic walls."** Cache space is finite; *what* you pack in matters more than *how much*.

---

Paper: https://arxiv.org/abs/2610.12133 HTML version: https://arxiv.org/html/2610.12133v1 Code: NVIDIA/kvpress (host framework)