On-Demand Attention: Language Models Know When to Recall — Paper Overview

Forum topic · 小凯 · 2026-09-20

Summary

On-Demand Attention (ODA) is a new local-first decoding method for efficient long-context inference, presented in arXiv paper 2609.20734 by Haibo Feng, Ruiqi Liang, Hanyang Peng, and Shiqi Yu. The authors observe that full-attention decoding reads the entire growing history at every step, regardless of its actual benefit to the next token prediction. They show that a pretrained model's decoding states already contain information predictive of this benefit before the global read occurs. ODA leverages this by training only a lightweight recall head that selectively invokes global attention when its predicted benefit changes during generation. Pretrained weights remain unchanged, and the complete historical KV cache stays available for future recall. The method is implemented with GPU-side conditional execution in vLLM, converting reduced global reads into practical decoding speedups for long contexts. Experiments on Qwen and Gemma model families, including hybrid attention backbones, show that selective recall recovers most of the performance lost to local attention while substantially reducing global reads, supporting a paradigm where pretrained models autonomously govern access to their own stored information.

Paper Overview

Field: NLP
Authors: Haibo Feng, Ruiqi Liang, Hanyang Peng, Shiqi Yu
Published: 2026-09-17
arXiv: 2609.20734

Abstract

Reasoning and agentic workloads increasingly demand efficient long-context inference. Yet full-attention decoding reads the growing history at every step, regardless of its benefit to the next prediction. The authors show that a pretrained model's decoding states already contain information predictive of this benefit, before the global read.

Building on this finding, they introduce On-Demand Attention (ODA), a local-first decoding method that uses a lightweight recall head to selectively invoke global attention as its predicted benefit changes during generation.

Key properties of ODA

Only the recall head is trained; pretrained weights remain unchanged.
The complete historical KV cache stays available for future recall.
GPU-side conditional execution is implemented in vLLM, translating reduced global reads into practical decoding speedups on long contexts.

Experimental results

Experiments on the Qwen and Gemma model families (including hybrid attention backbones) show that selective recall recovers most of the performance lost when using local attention alone, while significantly reducing global reads.

These findings support a long-context inference paradigm in which pretrained models autonomously guide access to the information they hold.

---

*Auto-collected on 2026-09-20*