You Only Index Once: Cross-Layer Sparse Attention with Shared Routing (CLSA) Explained

Forum topic · 小凯 · 2026-06-07

Summary

A Chinese forum post on zhichai.net explains a Microsoft Research Asia paper, 'You Only Index Once: Cross-Layer Sparse Attention with Shared Routing' (CLSA, arXiv:2606.015xx, June 2026, authors Yutao Sun, Yanqi Zhang, Li Dong). Building on YOCO (You Only Cache Once), which shares a single KV cache across all layers, CLSA extends the idea to token routing: a dedicated indexer computes token-level top-k selection only once in the first layer, and all subsequent layers reuse the shared index instead of recomputing routing per layer. This avoids the usual trade-off between block-sparse attention (fast but coarse) and token-sparse attention (accurate but routing-heavy). Reported results on 128K-token contexts include 7.6x faster decoding and 17.1x higher throughput versus standard attention, with accuracy losses typically under 1% on long-context QA, passage retrieval, and code completion benchmarks. Ablations attribute roughly 2.5x speedup to YOCO's KV sharing and a further ~3x to sparse routing, with cross-layer index sharing essential to keeping routing overhead low. The post discusses implications for real-time long-context applications, hardware design, and combination with MoE architectures.

Overview

A zhichai.net forum post introduces "You Only Index Once: Cross-Layer Sparse Attention with Shared Routing" (CLSA), a Microsoft Research Asia paper by Yutao Sun, Yanqi Zhang, and Li Dong (arXiv:2606.015xx, posted 2026-06-07, cs.CL/cs.AI/cs.LG). The post frames the work as a solution to the long-context dilemma of LLMs: attention scales quadratically (O(n²)) with sequence length, and KV cache memory grows linearly, making 128K-token inference slow and expensive.

Background: Attention and Its Limits

Attention computes Attention(Q, K, V) = softmax(QK^T / √d_k) × V — similarity between queries and keys weights the values.
At 128K tokens this implies billions of operations plus a prohibitive KV cache, motivating sparse attention.
Existing approaches trade off quality and overhead:
Block-sparse: fast and parallelizable, but coarse — may miss critical individual tokens.
Token-sparse (top-k): precise, but routing requires scoring all tokens, adding heavy per-layer overhead.

CLSA: Sharing the Index Across Layers

CLSA builds on YOCO (You Only Cache Once), which shares a single KV cache across all layers (reducing cache memory to 1/N for an N-layer model). CLSA's insight: if all layers share the same KV cache, they face the same routing question — so why recompute it?

Key components: 1. Indexer — in the first layer, computes token importance scores and top-k indices once. 2. Selector — every subsequent layer uses the shared index to gather KV values, without recomputing top-k. 3. Shared module — keeps index and cache consistent across layers.

This amortizes routing cost across all layers (up to 12x savings for a 12-layer model) while retaining token-level precision.

Key Results

| Metric | Result | |---|---| | Decoding speedup (128K context) | 7.6x | | Overall throughput | 17.1x | | KV cache memory | Significantly reduced | | Accuracy loss | Typically < 1% |

Benchmark examples: short-context QA 84.8% vs 85.2% baseline; long-context QA (128K) 78.1% vs 78.6%; passage retrieval 91.7% vs 92.3%; code completion 88.0% vs 88.4%.

Comparison: CLSA achieves 7.6x speedup with ~0.5% accuracy loss and low routing overhead, versus 8–10x/2–5% for block-sparse and 3–5x/0.5–1% with high routing overhead for token-sparse.

Ablations

Full CLSA: 7.6x, 84.8%
Without cross-layer shared index (per-layer recompute): 3.2x
Without YOCO KV sharing: 2.1x
YOCO only (no sparsity): 2.5x, 85.0%
Standard attention: 1.0x, 85.2%

YOCO's KV sharing contributes ~2.5x; sparse routing adds ~3x more; shared indexing is essential to avoid routing overhead erasing the gains.

Implications Discussed

Practical long-context AI: lower cost and latency enable legal document analysis, medical record synthesis, book-length research, and interactive use cases.
Architectural shift: from per-layer independence toward cross-layer sharing of core structures, analogous to shared DNA with specialized expression.
Hardware: reduced memory-bandwidth pressure; possible future dedicated routing/top-k units in AI accelerators.
Synergy with MoE: MoE sparsifies across layers (experts); CLSA sparsifies across sequence — combinable for multiplicative efficiency.

Conclusion

The post closes by framing CLSA's "index once" philosophy as an art of selective memory: intelligence lies not in storing everything, but in knowing how to find what matters — likening it to a central catalog for Borges' Library of Babel.