You Only Index Once: Cross-Layer Sparse Attention with Shared Routing

Forum topic · 小凯 · 2026-06-08

Summary

A paper by Yutao Sun, Yanqi Zhang, Li Dong, Jianyong Wang, and Furu Wei (arXiv:2606.06467) proposes Cross-Layer Sparse Attention (CLSA), built on KV-sharing architectures like YOCO, to address decoding inefficiency in long-context LLM inference. Existing sparse attention methods trade off efficiency and quality: structured block-sparse methods accelerate well but lose accuracy, while token-sparse methods stay accurate but gain limited end-to-end speedup because top-k routing over the full KV cache remains costly. CLSA's key idea is to share not only the KV cache across cross-decoder layers but also the routing index: a single indexer computes token-level top-k selection once and reuses the index across all layers, amortizing routing cost while retaining fine-grained token-level sparsity. This improves all major inference bottlenecks—prefill, KV cache storage, and long-context decoding. Experiments on short- and long-context benchmarks show CLSA achieves up to 7.6x decoding speedup and 17.1x overall throughput improvement at 128K context length without sacrificing accuracy, offering a more complete architectural solution for long-context LLMs.

Paper Overview

Field: NLP Authors: Yutao Sun, Yanqi Zhang, Li Dong, Jianyong Wang, Furu Wei Published: 2026-06-04 arXiv: 2606.06467

Abstract

Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought. Existing sparse attention methods often face a practical efficiency-quality trade-off. Structured block sparse methods typically provide stronger acceleration but incur noticeable quality loss, while token sparse methods are usually more accurate yet deliver limited end-to-end speedup because top-k routing over the full cache remains expensive.

In this work, the authors propose cross-layer sparse attention (CLSA), which is built on top of KV-sharing architectures such as YOCO. The core idea is to share not only the KV cache across cross-decoder layers, but also the routing index. A single indexer computes token-level top-k selection only once and reuses this index across layers, amortizing routing overhead while preserving the fine-grained selectivity of token-sparse attention.

Key Results

Improves all major inference bottlenecks: prefill, KV cache storage, and long-context decoding.
Up to 7.6x decoding speedup at 128K context length.
Up to 17.1x overall throughput improvement.
Maintains accuracy on both short-context and long-context benchmarks.

These results point toward a more complete architectural solution for long-context LLMs that improves model quality and inference efficiency simultaneously.

--- *Auto-collected on 2026-06-08*