Using Speculative Decoding Attention Scores to Guide Sparse Attention in Large Models

Forum topic · 小凯 · 2026-05-19

Summary

This post from zhichai.net discusses STS, a method by Xu, Yu, Wu, and Xie that reuses attention scores produced by a small model during speculative decoding to construct dynamic sparse attention masks for a large language model. The core assumption is that tokens the small model attends to are also important to the large model, validated by high overlap in top-K important tokens. Since the small model already computes these scores during speculative decoding's forward pass, the extra overhead is nearly zero. On the NarrativeQA benchmark, STS achieves a 2.67x speedup at roughly 90% sparsity with almost no accuracy loss, reaching higher sparsity than prior sparse attention methods at equal accuracy. The author also raises open questions about attention alignment consistency across tasks, failure modes when the small model misses critical tokens, and applicability when speculative decoding is not part of the deployment setup. References include Leviathan et al. (ICML 2023) on speculative decoding and Child et al. (NeurIPS 2019) on sparse transformers.

LLM inference's biggest cost is attention computation. When context reaches the million-token scale, every token must attend to all previous tokens — O(n²) complexity that even the most advanced hardware cannot bear. Sparse attention is the solution: not every token needs to attend to all history; most tokens' influence can be safely ignored. But how do you know which tokens can be ignored?

The approach by Xu, Yu, Wu, and Xie is clever: let a small model make that judgment for the large model. Speculative decoding was originally designed to accelerate generation — a small model first drafts candidate tokens, and the large model verifies them. This paper takes the attention scores the small model produces during speculative decoding and uses them as the basis for sparsifying the large model's attention.

Core assumption: tokens the small model considers important are also important to the large model. This assumption is validated in experiments — the small model's attention distribution overlaps highly with the large model's attention distribution on top-K important tokens.

Based on this, at each attention head and each token position, the small model's attention scores are used to dynamically construct a sparse mask — which tokens deserve full attention computation, and which can be skipped entirely. Because the small model already computes these scores in speculative decoding's forward pass, the additional overhead is close to zero.

On the NarrativeQA benchmark, STS achieves a 2.67x speedup at approximately 90% sparsity with almost no loss in accuracy. Compared with existing sparse attention methods, STS reaches higher sparsity at equivalent accuracy.

Open questions: Is attention alignment between the small and large model stable across different tasks? If the small model makes a serious mistake (missing a critical token), will the large model follow it into error? The speculative decoding framework is not default in all deployment scenarios — if you don't use speculative decoding, is STS's additional overhead still worth it?

---

References

1. Xu, C., Yu, J., Wu, Y., & Xie, Y. (2026). *STS: Efficient Sparse Attention with Speculative Token Sparsity*. arXiv:2605.15508 [cs.LG].

2. Leviathan, Y., et al. (2023). *Fast Inference from Transformers via Speculative Decoding*. ICML.

3. Child, R., et al. (2019). *Generating Long Sequences with Sparse Transformers*. NeurIPS.

Using Speculative Decoding Attention Scores to Guide Sparse Attention in Large Models

Summary

Tags