Imagine searching a huge library and, every time you read a new book, re-searching from scratch for the relevant sections. That is essentially the predicament of current LLM long-context inference.
Now someone has proposed an idea so simple it makes you slap your forehead: index once—why re-index at every layer?
The "Three Mountains" of Long Context
Modern LLMs face three efficiency bottlenecks in long-context inference:
1. Slow prefill: the longer the input, the slower the first processing pass. 2. Large KV cache: key-value pairs must be stored for every layer, straining memory. 3. Slow decoding: generating each new token requires searching all history for relevant information.
Existing sparse attention methods target the third problem but face a dilemma:
- Block-sparse attention (e.g., MoBA): regular structure, GPU-friendly, large speedups—but coarse granularity leads to noticeable quality loss.
- Token-sparse attention (e.g., Quest): fine granularity and better quality—but top-k routing must be run per layer, and the routing itself is slow.
- Self-Decoder: processes the input and produces a shared KV cache.
- Cross-Decoder: retrieves information from the shared KV cache to generate output.
- 7.6x decoding throughput improvement (vs. a Transformer baseline)
- 17.1x end-to-end throughput improvement (including prefill speedups)
- Substantially reduced KV cache footprint (shared KV saves space by itself)
The core contradiction: routing decisions are expensive, yet every layer independently makes the same routing decision.
Cross-Layer Shared Routing: One Index, Reused by the Whole Model
YOIO (You Only Index Once)'s core idea is extremely simple: since multiple layers read the same KV cache, the routing index should also be computed only once.
Specifically, YOIO builds on the YOCO architecture, which splits the model into two parts:
YOIO adds a lightweight "query-aware indexer" on top of the Self-Decoder that computes, in one pass, which KV positions each query token should attend to. All Cross-Decoder layers then share this index.
It's like a library with a professional indexer: readers don't re-search the catalog every time they enter a new reading room—they just follow the index's signposts.
Why Doesn't Sharing the Index Hurt Quality?
This is the key question. Intuitively, different layers should attend to different things—doesn't a shared index apply a one-size-fits-all solution?
The paper's detailed analysis of attention patterns finds: under a shared-KV-cache architecture, routing decisions across layers are highly consistent. Because all layers read the same memory, they need the same information sources—they differ only in how they process the information (some layers retrieve, others transform).
Experiments confirm this: on both short- and long-context benchmarks, the method performs nearly losslessly compared to dense attention baselines.
How Big Are the Efficiency Gains?
At 128K context length:
A Deeper Insight
YOIO's significance goes beyond "yet another acceleration method." It reveals an important architectural insight: in a Transformer's multi-layer structure, "where to find information" and "how to process it" are independent decisions—the former can be shared, the latter should stay independent.
This mirrors human information processing: when reading an article, your eye movement patterns (where to look) are similar across reading stages, but how your brain processes what it sees varies by stage.
YOIO performs the expensive "where to look" decision once and reuses it across all layers, while the "how to process" part—which needs diversity—remains independent per layer. This separation captures the optimal balance between efficiency and quality.
Index once, share across layers. Simple, but effective.
---
📄 Paper: You Only Index Once: Cross-Layer Sparse Attention with Shared Routing 💻 Related code: github.com/THUDM/IndexCache