You Only Index Once: Cross-Layer Sparse Attention with Shared Routing (CLSA)

Forum topic · 小凯 · 2026-06-08

Summary

Cross-layer sparse attention (CLSA) is a new sparse attention architecture for long-context large language models, built on KV-sharing designs such as YOCO. Unlike prior sparse attention methods that trade off quality for speed, CLSA shares not only the KV cache across cross-decoder layers but also the routing index. A single indexer computes token-level top-k selection once and reuses that index across all layers, amortizing routing overhead while retaining the fine-grained selectivity of token-sparse attention. This simultaneously improves all major inference bottlenecks: prefilling, KV cache storage, and long-context decoding. Experiments on short- and long-context benchmarks show CLSA is both accurate and efficient, achieving up to 7.6x decoding speedup and 17.1x overall throughput improvement at a 128K context length. The work by Yutao Sun, Yanqi Zhang, Li Dong, Jianyong Wang, and Furu Wei (arXiv 2606.06467) points toward a more complete architectural solution for long-context LLMs that improves both model quality and inference efficiency, particularly in reasoning-heavy workloads with long chains of thought.

Paper Overview

Field: NLP
Authors: Yutao Sun, Yanqi Zhang, Li Dong, Jianyong Wang, Furu Wei
Date: 2026-06-04
arXiv: 2606.06467

Abstract

Long-context inference in modern LLMs is increasingly constrained by decoding efficiency, especially in reasoning-heavy settings where models generate long intermediate chains of thought. Existing sparse attention methods often face a practical efficiency-quality trade-off. Structured block sparse methods typically provide stronger acceleration but incur noticeable quality loss, while token sparse methods are usually more accurate yet deliver limited end-to-end speedup because top-k routing over the full cache remains expensive.

In this work, the authors propose cross-layer sparse attention (CLSA), built on top of KV-sharing architectures such as YOCO. The core idea is to share not only the KV cache across cross-decoder layers, but also the routing index. A single indexer computes token-level top-k selection once and reuses the index across layers, amortizing routing overhead while preserving the fine-grained selectivity of token-sparse attention.

The resulting architecture simultaneously improves all major inference bottlenecks, including prefilling, KV cache storage, and long-context decoding. Experiments on short-context and long-context benchmarks show that CLSA is both accurate and efficient, achieving up to 7.6x decoding speedup and 17.1x overall throughput improvement at a 128K context length. These results point to a more complete architectural solution for long-context LLMs that improves both model quality and inference efficiency.

Key Contributions

1. Shared routing index: A single indexer performs top-k token selection once and shares the index across all cross-decoder layers, eliminating redundant per-layer routing costs. 2. Amortized token sparsity: Retains fine-grained token-level selectivity of token-sparse attention without the expensive per-layer top-k routing over the full cache. 3. End-to-end speedups: Improvements across prefilling, KV cache storage, and decoding; up to 7.6x decoding acceleration and 17.1x overall throughput gain at 128K context length. 4. Compatibility with KV-sharing architectures: Designed on top of frameworks like YOCO.

*Auto-collected on 2026-06-08.*