You Only Index Once: Cross-Layer Sparse Attention with Shared Routing (CLSA) Explained

Forum topic · 小凯 · 2026-06-07

Summary

A detailed Chinese forum post explains CLSA (Cross-Layer Sparse Attention), a sparse attention method building on YOCO (You Only Cache Once) from Microsoft Research Asia. CLSA's core insight is that when all transformer layers share a single KV cache, they also share the same routing problem, so the expensive token-level top-k selection should be computed once by an indexer in the first layer and reused across all layers. This 'index once, share everywhere' design amortizes routing overhead while retaining token-level sparsity precision. Reported results on 128K-token contexts include 7.6x decoding speedup, 17.1x throughput improvement, and significant KV cache memory reduction, with less than 1% accuracy loss on benchmarks such as long-context QA and passage retrieval. Ablations show YOCO's KV sharing contributes ~2.5x speedup, sparse attention another ~3x, and the shared index itself is critical (3.2x vs 7.6x without it). The post argues CLSA enables practical long-context LLM applications (legal, medical, academic analysis), signals a paradigm shift toward cross-layer sharing, and could compose with MoE for 'double sparsification' across layer and sequence dimensions.

Overview

This forum post on zhichai.net discusses the paper "You Only Index Once: Cross-Layer Sparse Attention with Shared Routing" by Yutao Sun, Yanqi Zhang, and Li Dong (Microsoft Research Asia), posted to arXiv (cs.CL, cs.AI, cs.LG) on 2026-06-07. The method, CLSA (Cross-Layer Sparse Attention), extends YOCO (You Only Cache Once) by sharing not just the KV cache across layers, but also the sparse attention routing index.

Background: The Long-Context Problem

Standard attention has O(n²) complexity; processing 128K tokens means billions of operations plus a KV cache that scales with context length, often exceeding GPU memory.
Existing sparse attention approaches face a trade-off:
Block-sparse attention is fast and parallelizable but coarse — it can miss important individual tokens.
Token-level sparse attention is precise (top-k selection) but the routing overhead is expensive, since similarity must be computed against all tokens.

CLSA's Core Idea: Share the Index Across Layers

YOCO already shares a single KV cache across all layers (reducing memory by ~1/12 for a 12-layer model). CLSA's insight: if all layers share the same KV cache, they share the same routing question — "which tokens matter?" — so the answer should be computed once:

1. Indexer (first layer): computes importance scores and token-level top-k selections. 2. Selector (all subsequent layers): reads from the KV cache using the shared index without recomputing top-k. 3. Shared module: keeps index and KV cache consistent across layers.

This combines block-sparse speed with token-sparse precision: routing cost is paid once instead of per-layer (a ~12x saving for a 12-layer model), while index storage remains tiny (just integer arrays).

Key Results (128K context)

7.6x decoding speedup — turning long-context generation from batch jobs into interactive tasks.
17.1x overall throughput improvement — same hardware serving ~17x more requests.
Significant KV cache memory reduction and improved prefill speed.
Accuracy loss is small (<1%): e.g., long-context QA 78.6% → 78.1%, passage retrieval 92.3% → 91.7%, short-context QA 85.2% → 84.8%, code completion 88.4% → 88.0%.

Ablation findings

| Configuration | Decoding speed | Accuracy | |---|---|---| | Full CLSA | 7.6x | 84.8% | | Without cross-layer shared index | 3.2x | 84.6% | | Without YOCO KV sharing | 2.1x | 84.5% | | Base YOCO (no sparsity) | 2.5x | 85.0% | | Standard attention | 1.0x | 85.2% |

YOCO's KV sharing contributes ~2.5x, CLSA's sparsity another ~3x, and the shared index is critical (3.2x vs 7.6x). Accuracy loss mainly comes from sparsification itself, not the sharing.

Implications Discussed

Practical long-context AI: affordable legal document analysis, medical record synthesis, book-length academic research, and business intelligence.
Architectural paradigm shift: from per-layer independence to cross-layer sharing of key structures (caches, routing), analogous to shared DNA with specialized expression.
Hardware: reduced memory-bandwidth pressure; potential for dedicated routing/top-k units in future AI chips.
Synergy with MoE: MoE sparsifies along the layer (expert) dimension; CLSA sparsifies along the sequence dimension — combining both could yield multiplicative efficiency gains.

Conclusion

The post frames CLSA's "you only index once" philosophy as an analogy to human selective memory: intelligence is not about remembering everything, but knowing how to find information — building a central index for AI's "Library of Babel."

References

1. Sun, Y., Zhang, Y., Dong, L., et al. "You Only Index Once: Cross-Layer Sparse Attention with Shared Routing." arXiv preprint arXiv:2606.015xx (2026). 2. Su, Y., et al. "YOCO: You Only Cache Once, Decoder-Decoder Architectures for Language Models." arXiv 2024. 3. Vaswani, A., et al. "Attention Is All You Need." NeurIPS 2017. 4. Child, R., et al. "Generating Long Sequences with Sparse Transformers." arXiv 2019. 5. Kitaev, N., Kaiser, Ł., Levskaya, A. "Reformer: The Efficient Transformer." ICLR 2020. 6. Beltagy, I., Peters, M.E., Cohan, A. "Longformer: The Long-Document Transformer." arXiv 2020. 7. Xiao, G., et al. "Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM." arXiv 2023. 8. Shazeer, N., et al. "Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer." ICLR 2017.