[论文] You Only Index Once: Cross-Layer Sparse Attention with Shared Routing

论文概要

研究领域: NLP
作者: Yutao Sun, Yanqi Zhang, Li Dong, Jianyong Wang, Furu Wei
发布时间: 2026-06-04
arXiv: 2606.06467

中文摘要

现代大语言模型的长上下文推理越来越受到解码效率的制约，尤其是在模型生成长中间思维链的推理密集型场景中。现有的稀疏注意力方法往往面临实际的效率-质量权衡。结构化块稀疏方法通常提供更强的加速效果，但会带来明显的质量损失；而 token 稀疏方法通常更准确，但由于在整个缓存上进行 top-k 路由仍然代价高昂，因此端到端加速有限。本文提出跨层稀疏注意力（CLSA），构建在 YOCO 等 KV 共享架构之上。核心思想是不仅在跨解码器层之间共享 KV 缓存，还共享路由索引。单个索引器只计算一次 token 级别的 top-k 选择，并在各层之间复用该索引，从而在分摊路由开销的同时保留 token 稀疏注意力的细粒度选择性。由此产生的架构同时改善了所有主要推理瓶颈，包括预填充、KV 缓存存储和长上下文解码。在短上下文和长上下文基准上的实验表明，CLSA 既准确又高效，在128K上下文长度下实现了最高7.6倍的解码加速和17.1倍的整体吞吐量提升。这些结果表明了一种更完整的长上下文大语言模型架构解决方案，能够同时提升模型质量和推理效率。

原文摘要

Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought. Existing sparse attention methods often face a practical efficiency-quality trade-off. Structured block sparse methods typically provide stronger acceleration but incur noticeable quality loss, while token sparse methods are usually more accurate yet deliver limited end-to-end speedup because top-k routing over the full cache remains expensive. In this work, we propose cross-layer sparse attention (CLSA), which is built on top of KV-sharing architectures such as YOCO. The core idea is to share not only the KV cache across cross-decoder layers, but also the routing index. A single indexer computes token-le...

自动采集于 2026-06-08

#论文 #arXiv #NLP #小凯