You are chatting with an AI about a million-token document. You ask: "What was the control group's sample size mentioned earlier?"
To answer that one question, every attention layer sweeps the entire million-token KV cache—even though the answer lives in three lines. It's like searching a library for one sentence by re-walking every bookshelf for every character you read.
This is not an exaggeration; it's the real cost of current Transformer inference. Attention was designed for "global vision," but in long-context settings, global vision becomes a global tax.
The Model Already Knows Where to Look
This paper asks a deceptively simple question: if the model can generate the correct answer, it already knows where the relevant information is—so why force it to re-scan the full text at every step?
The authors, Namgyu Ho et al. (KAIST + Google DeepMind), propose Declarative Attention (DA): the model actively "declares" in its chain of thought which context regions it currently needs. The inference engine parses these declarations like tool calls, switches attention modes, and directly skips KV cache reads that aren't needed.
Three Attention Modes
DA splits generation into three modes the model switches between itself:
<global>: full attention over the entire context, for tasks requiring global reasoning.<focus>: narrowed to a specific region, reading only that region's KV cache—for looking up specific earlier details.<local>: only recent output, no history—used for fluent generation once the answer is found.- DA currently applies only to global attention layers; local attention layers are already cheap.
- The small accuracy drop may amplify on tasks requiring precise global reasoning (e.g., extreme needle-in-a-haystack variants).
- Zero-shot DA is a lower bound; post-training can further narrow the accuracy gap.
It mirrors human reading strategy: skim globally, focus on a passage, then write while staring at your draft. The model doesn't need to be taught these strategies—zero-shot prompting suffices.
The Numbers
Tested on 15 long-context tasks with two off-the-shelf models:
| Model | Attention token reduction | Accuracy drop | |------|-------------------|------------| | Gemma-4-31B | 52.0% | 1.27pp | | Qwen-3.6-27B | 31.1% | 2.75pp |
Half the attention cost is cut for a 1–3 percentage point accuracy loss. That gap shrinks with model scale—the stronger the model, the smaller DA's cost.
Crucially, the token savings rate is roughly independent of model size. DA is not a "small-model compromise" but an efficiency lever that holds across scale.
Why This Matters
Current long-context efficiency work follows two main routes:
1. KV cache eviction: drop "unimportant" tokens by heuristic scores—but "unimportant" is defined by external rules, not the model. 2. Sparse attention: dynamically select attention targets during decoding—but selection still comes from an external scorer with O(N) scan cost.
DA takes a third path: let the model declare what it needs. This bypasses the external-scorer bottleneck, because the model already implicitly knows where relevant information is when generating the answer—DA merely makes that implicit knowledge explicit.
Isomorphism with Tool Calls
DA is structurally isomorphic to function calling. In tool calls, the model declares "I need to call this function" and the engine executes it. In DA, the model declares "I need to attend to this context" and the engine adjusts the attention mask and skips the rest.
Both share one insight: the model already knows what to do; it just needs a channel to express it. Tool calls let the model express which external tools it needs; DA lets it express which internal memory it needs.
System-2 KV Cache Offloading
The paper also sketches a more aggressive idea: DA as a foundation for "system-2 sparse attention." When the model enters <focus> mode, unneeded KV cache can be offloaded to cheaper storage (e.g., CPU memory), reloaded only when <global> is declared.
Attention shifts from "fully resident at every step" to "loaded on demand"—like adding a model-declared tier to the L1/L2/L3 cache hierarchy for KV cache.
Limitations
The authors are candid about constraints:
My Take
DA's insight pairs interestingly with "judgment-gate decoupling": there, the model internally knows an action is wrong but the action gate never consults the judgment module. DA shows the mirror image: the model internally knows where to look, but traditional attention never lets it express that knowledge.
Both point to the same design principle: models hold a lot of implicit knowledge; the key isn't making the model smarter but giving it a channel to turn implicit knowledge into explicit control. Tool calls externalize tool invocation; DA externalizes internal attention routing.
As model scale and context windows grow, "letting the model decide where to look" will shift from an optimization option to a necessity—full-text scanning over a million tokens is physically unsustainable on HBM bandwidth alone. DA offers a zero-training transition path, and a post-trained version may flatten the accuracy gap further.
---
Paper: arXiv:2609.02737 Authors: Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, et al. Code: No standalone repository provided; the DA protocol can be implemented on any inference engine supporting tool calls.