You are chatting with an AI about a million-token document. You ask: "What was the sample size of the control group in that experiment mentioned earlier?"
To answer that one question, every attention layer must scan the entire million-token KV cache—even though the answer lives in just three lines. It's like searching a library where, for every character you read, you have to walk past every bookshelf again.
That's not hyperbole; that's the real cost of Transformer inference today. Attention was designed for "global vision," but in long-context settings, global vision becomes a global tax.
The model already knows where to look
The paper (arXiv:2609.02737) asks a question so simple it seems suspect: if the model can produce the correct answer, it already knows where the relevant information is—so why force it to rescan everything at every step?
The authors, Namgyu Ho et al. (KAIST + Google DeepMind), propose Declarative Attention (DA): the model actively "declares" which context regions it needs within its chain of thought. The inference engine parses these declarations like tool calls, switching attention modes and skipping unnecessary KV cache reads.
Three attention modes
DA splits generation into three modes that the model switches between itself:
<global>: full attention over the entire context, for tasks requiring global reasoning.<focus>: attention restricted to a specific region—only that region's KV cache is read, ideal for looking back at specific details.<local>: attention only on recent output, no history read—used for fluent generation once the answer is found.- DA currently only applies to global attention layers; local attention layers are already cheap.
- The small accuracy drop could amplify on tasks requiring precise global reasoning (e.g., extreme needle-in-a-haystack variants).
- Zero-shot DA is a lower bound; post-training could further close the accuracy gap.
It mirrors human reading strategy: skim globally, locate a paragraph with focus, and write the answer staring only at your draft. The model doesn't need to be taught these strategies—zero-shot prompting suffices.
The numbers
Tested on 15 long-context tasks with two off-the-shelf models:
| Model | Attention token reduction | Accuracy drop | |-------|---------------------------|---------------| | Gemma-4-31B | 52.0% | 1.27pp | | Qwen-3.6-27B | 31.1% | 2.75pp |
Half of the attention cost is eliminated with only a 1–3 point accuracy drop. Moreover, the gap shrinks as model scale grows—the stronger the model, the smaller DA's cost.
Crucially, the token savings rate is largely independent of model size. DA is not a "compromise for small models" but an efficiency lever that holds its value across scales.
Why this matters
Current long-context efficiency work follows two paths:
1. KV cache eviction: drop "unimportant" tokens by heuristic scores. But what counts as unimportant is defined by external rules, not the model. 2. Sparse attention: dynamically select attention targets during decoding—but the selection logic is still an external scorer with O(N) scan cost.
DA takes a third path: let the model declare what it needs. This bypasses the external-scorer bottleneck, because the model already implicitly knows where the relevant information is while generating—DA simply turns that implicit knowledge into an explicit declaration.
Isomorphism with tool calls
DA's design is structurally isomorphic to function calling. In tool calls, the model declares "I need to call this function" and the engine executes it. In DA, the model declares "I need this context region" and the engine adjusts the attention mask and skips the rest.
Both share one insight: the model already knows what to do—it just needs a channel to express it. Tool calls let the model express which external tools it needs; DA lets it express which internal memory it needs.
System-2 KV cache offloading
The paper sketches a more ambitious vision: DA as the basis for "system-2 sparse attention." When the model enters <focus> mode, unneeded KV cache can be offloaded to cheaper storage (e.g., CPU memory) and reloaded only when <global> is declared.
This turns attention from "fully resident at every step" into "loaded on demand." Like the L1/L2/L3 hierarchy in CPU caches, DA adds a model-declared tier to the KV cache.
Limitations
The authors are candid about limits:
My take
DA's insight pairs interestingly with "judgment-gate decoupling." There, the model internally knows the answer is wrong but the action gate never consults the judgment module. DA shows the mirror image: the model internally knows where to look, but traditional attention doesn't let it express that knowledge.
Both point to the same design principle: models hold a lot of implicit knowledge—the key isn't making the model smarter, but giving it a channel to turn implicit knowledge into explicit control. Tool calls externalized "tool invocation"; DA externalizes "internal attention routing."
As model scale and context windows keep growing, "letting the model decide where to look" will shift from an optimization option to a necessity. Full scans over a million tokens are physically unsustainable—HBM bandwidth alone can't keep up. DA offers a zero-training-cost transitional solution, and a post-trained version may further erase the accuracy gap.
---
Paper: arXiv:2609.02737 Authors: Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, et al. Code: No standalone repository provided; the DA protocol can be implemented in any inference engine supporting tool calls.