You are chatting with an AI about a million-token document. You ask: "What was the control group's sample size in the experiment mentioned earlier?"
To answer that single question, every attention layer must sweep the entire million-token KV cache—even though the answer sits in three lines. It's like searching a library for one sentence by re-walking every bookshelf for every word you read.
This is not an exaggeration; it is the real cost of current Transformer inference. Attention was designed for "global vision," but in long-context settings global vision becomes a global tax.
The Model Already Knows Where to Look
The paper asks a question so simple it seems suspicious: if the model can generate correct answers, it already knows where the relevant information is—so why force it to re-scan everything at every step?
The authors (Namgyu Ho et al., KAIST + Google DeepMind) propose Declarative Attention (DA): the model actively "declares" in its chain of thought which context regions it currently needs. The inference engine parses these declarations like tool calls, switches attention modes, and skips KV cache reads that aren't needed.
Three Attention Modes
DA splits generation into three modes that the model switches between itself:
<global>: full attention over the entire context, for global reasoning.<focus>: attention restricted to a specific region, reading only that region's KV cache—for looking up specific details mentioned earlier.<local>: attention only on recent output, no history—for fluent generation once the answer is found.- DA currently only applies to global attention layers; local attention layers are already cheap.
- The small accuracy drop may amplify on tasks requiring precise global reasoning (extreme needle-in-a-haystack variants).
- Zero-shot DA is a lower bound; post-training could further shrink the accuracy gap.
It mirrors human reading strategy: skim everything with global, locate a passage with focus, and write the answer while staring at your draft with local. The model doesn't need to be taught these strategies—zero-shot prompting suffices.
The Numbers
Tested on 15 long-context tasks with two off-the-shelf models:
| Model | Attention token reduction | Accuracy drop | |-------|---------------------------|---------------| | Gemma-4-31B | 52.0% | 1.27pp | | Qwen-3.6-27B | 31.1% | 2.75pp |
Half the attention overhead is cut for a 1–3 percentage-point accuracy cost. The gap shrinks with model scale—the stronger the model, the lower DA's cost.
Crucially, the token savings ratio is roughly independent of model size. DA is not a "small-model compromise" but an efficiency lever whose effectiveness persists at scale.
Why It Matters
Current long-context efficiency work follows two main paths:
1. KV cache eviction: drop "unimportant" tokens by heuristic scores—but "unimportant" is defined by external rules, not the model. 2. Sparse attention: dynamically select attention targets during decoding—but the selection logic is still an external scorer, still O(N) scanning.
DA takes a third path: let the model declare what it needs. This bypasses the external-scorer bottleneck, because the model already implicitly knows where relevant information is while generating—DA just turns that implicit knowledge into an explicit declaration.
Isomorphism with Tool Calls
DA is structurally analogous to function calling. In tool calls, the model declares "I need to invoke this function" and the engine executes it. In DA, the model declares "I need to attend to this context" and the engine adjusts the attention mask and skips the rest.
Both share the same insight: the model already knows what to do; it just needs a channel to express it. Tool calls let the model express which external tools it needs; DA lets it express which internal memory it needs.
System-2 KV Cache Offloading
The paper sketches a more radical idea: DA as a foundation for "system-2 sparse attention." When the model enters <focus> mode, unneeded KV caches can be offloaded to cheaper storage (e.g., CPU memory), reloaded only when the model declares <global>.
This turns attention from "fully resident at every step" into "loaded on demand." Like the L1/L2/L3 hierarchy of CPU caches, DA adds a model-declared tier to the KV cache hierarchy.
Limitations
The authors honestly list restrictions:
My Take
DA's insight pairs interestingly with "judgment-gate decoupling": there, the model internally knows the answer is wrong but the action gate never consults the judgment module. DA shows the flip side: the model internally knows where to look, but traditional attention never lets it express that knowledge.
Both point to one design principle: models hold lots of implicit knowledge; the key is not making them smarter but giving them a channel to turn implicit knowledge into explicit control. Tool calls externalize tool invocation; DA externalizes internal attention routing.
As model scale and context windows keep growing, "letting the model decide where to look" will shift from optimization option to necessity. Full-context scans over a million tokens are physically unsustainable—HBM bandwidth alone can't keep up. DA offers a zero-training-cost bridge, and a post-trained version may close the accuracy gap further.
---
Paper: arXiv:2609.02737 Authors: Namgyu Ho, Huzama Ahmad, Woosung Koh, Se-Young Yun, Tal Schuster, et al. Code: no standalone repository provided; the DA protocol can be implemented on any inference engine supporting tool calls