You Only Index Once: Shared Routing Across Layers for Efficient Long-Context Inference

Forum topic · 小凯 · 2026-06-05

Summary

YOIO (You Only Index Once) is a sparse attention technique that accelerates long-context LLM inference by computing token routing decisions only once and sharing them across all decoder layers. Built on the YOCO (You Only Cache Once) architecture with a shared KV cache, YOIO adds a lightweight query-aware indexer above the self-decoder; all cross-decoder layers then reuse this single index instead of performing independent top-k routing. The key insight is that layers reading the same shared KV cache make highly consistent routing decisions, so sharing an index causes near-zero quality loss while avoiding per-layer routing overhead. Compared with prior approaches like block-sparse attention (MoBA, GPU-friendly but coarse) and token-sparse attention (Quest, fine-grained but slow routing), YOIO balances quality and speed. At 128K context length, it reportedly achieves 7.6x decode throughput improvement and 17.1x end-to-end throughput over a Transformer baseline, with substantially reduced KV cache memory and near-lossless benchmark performance. Paper: arXiv 2606.06467; code: github.com/THUDM/IndexCache.

The problem: long-context inference bottlenecks

Modern large models face three efficiency bottlenecks during long-context inference:

1. Slow prefill — the longer the input, the slower the first processing pass 2. Large KV caches — key-value pairs stored for every layer strain memory 3. Slow decoding — every generated token requires searching all history for relevant information

Existing sparse attention methods target the third problem but face a dilemma:

Block-sparse attention (e.g., MoBA): regular structure, GPU-friendly, strong speedups — but coarse granularity causes noticeable quality loss
Token-sparse attention (e.g., Quest): fine-grained and high quality — but each layer must run its own top-k routing, and the routing itself is slow

The core contradiction: routing decisions are expensive, yet every layer independently makes essentially the same routing decision.

The idea: one index, reused by all layers

YOIO's core idea is strikingly simple: since multiple layers read the *same* KV cache, the routing index should be computed only once.

YOIO builds on the YOCO architecture, which splits the model into:

Self-Decoder: processes input and generates a shared KV cache
Cross-Decoder: retrieves information from the shared KV cache to produce output

YOIO adds a lightweight query-aware indexer on top of the self-decoder that computes, in one pass, which KV positions each query token should attend to. All cross-decoder layers then share this single index.

Why sharing the index doesn't hurt quality

Intuitively, different layers attend to different content — wouldn't a shared index be a blunt compromise?

The paper's attention-pattern analysis shows that in shared-KV-cache architectures, routing decisions across layers are highly consistent. The layers read the same memory and need the same information sources; they differ only in how they process the retrieved information (some layers retrieve, others transform).

Experiments confirm this: the resulting CLSA model is nearly lossless compared to dense attention baselines on both short- and long-context benchmarks.

Efficiency gains

At 128K context length:

7.6x decode throughput vs. the Transformer baseline
17.1x end-to-end throughput (including prefill speedup)
Substantially reduced KV cache footprint (the shared cache itself saves memory)

These gains come without sacrificing quality — benchmark performance is essentially unchanged.

The deeper insight

YOIO is more than another acceleration trick. It reveals an architectural insight: in Transformer stacks, "where to find information" and "how to process it" are separable decisions — the former can be shared across layers, while the latter stays layer-specific.

One index, many layers. Simple, but effective.

---

📄 Paper: You Only Index Once: Cross-Layer Sparse Attention with Shared Routing 💻 Code: github.com/THUDM/IndexCache