On-Demand Attention: Language Models Know When to Recall

Forum topic · 小凯 · 2026-09-20

Summary

On-Demand Attention (ODA) is a new local-first decoding method for efficient long-context inference in large language models, proposed by Feng et al. in arXiv paper 2609.20734. The key insight is that a pretrained model's decoding states already contain information predictive of how much a global attention read will benefit the next token prediction, before that read occurs. ODA exploits this by adding a lightweight recall head that selectively invokes global attention based on its predicted benefit during generation. Only the recall head is trained, keeping pretrained weights unchanged, and the complete historical KV cache remains available for future recall. The authors implement GPU-side conditional execution in vLLM, converting reduced global reads into practical decoding speedups for long contexts. Experiments on the Qwen and Gemma model families, including hybrid attention backbones, show that selective recall recovers most of the performance lost by local-only attention while substantially reducing global reads. The work suggests a paradigm where pretrained models autonomously decide when to access their stored context, benefiting reasoning and agentic workloads with long histories.

Paper Overview

Field: NLP
Authors: Haibo Feng, Ruiqi Liang, Hanyang Peng, Shiqi Yu
Published: 2026-09-17
arXiv: 2609.20734

Motivation

Reasoning and agentic workloads increasingly demand efficient long-context inference. However, full-attention decoding reads the entire growing history at every step, regardless of whether that history actually benefits the next-token prediction.

Key Idea

The authors show that a pretrained model's decoding states already contain information predictive of the benefit of a global attention read — before the read happens. Building on this finding, they propose On-Demand Attention (ODA), a local-first decoding method:

A lightweight recall head selectively invokes global attention as its predicted benefit changes during generation.
Only the recall head is trained; pretrained weights stay unchanged.
The complete historical KV cache is always available for future recall.

Implementation

The authors implement GPU-side conditional execution in vLLM, translating the reduced number of global reads into practical decoding speedups for long contexts.

Results

Experiments on the Qwen and Gemma model families (including hybrid attention backbones) show that selective recall recovers most of the performance lost by local-only attention while significantly reducing global reads.

Takeaway

These findings support a new paradigm for long-context inference: pretrained models can autonomously decide when to access the information they hold, rather than reading all history at every decoding step.

---

Original paper: On-Demand Attention: Language Models Know When to Recall (arXiv:2609.20734)