Not All Thoughts Need HBM: Semantics-Aware KV Cache Tiering for Reasoning LLMs

Forum topic · 小凯 · 2026-05-18

Summary

Reasoning LLMs generate thousands of chain-of-thought tokens whose KV cache must fit in scarce GPU HBM. Common token eviction approaches fail badly on reasoning tasks: removing half the cache can drive accuracy to zero. A paper by Yuan, Shen, and Zhang (arXiv:2605.09490) proposes a semantics-aware memory hierarchy that tiers tokens across four levels—HBM, DDR, compressed, and evicted. Low-importance tokens are moved to CPU memory instead of being destroyed, and are prefetched at full precision before each attention computation, as if they never left the GPU. The core finding: accuracy depends only on how many tokens are permanently evicted, not on how many stay in HBM. Across 7B–32B models and four benchmarks, evicting just 3% of tokens preserves 91% of full-cache accuracy; on a 14B model it matches the uncompressed baseline (90% vs 86%) while halving HBM usage. Open questions remain around prefetch latency, since the reported 5–7% transfer overhead likely assumes ample CPU-GPU bandwidth and may degrade under concurrent multi-user workloads.

Reasoning LLMs generate thousands of chain-of-thought tokens, and every token's KV cache must live in precious GPU HBM. The mainstream approach is to evict unimportant tokens from the cache—but for reasoning tasks this is disastrous: removing half the cache can reduce accuracy to zero.

Yuan, Shen, and Zhang (arXiv:2605.09490) asked a different question: does every token really need to live in HBM? Could some tokens reside elsewhere?

Key points

Their semantics-aware memory hierarchy tiers tokens into four levels: HBM, DDR, compressed, and evicted.
Low-importance tokens are not destroyed—they are moved to CPU memory (DDR).
Before each attention computation, they are prefetched back at full precision, as if they had never left the GPU.

Core finding

Accuracy depends only on how many tokens you permanently discard (the eviction ratio), not on how many tokens remain in HBM. Across 7B–32B models and four benchmarks:

Evicting only 3% of tokens preserves 91% of full-cache accuracy.
On a 14B model, the method matches the uncompressed baseline (90% vs 86%) while halving HBM occupancy.

Open questions

The impact of prefetch latency is unclear—fetching tokens from DDR takes extra time. The paper reports transfer overhead of only 5–7%, but this likely holds only when CPU–GPU bandwidth is sufficient. Under concurrent multi-user workloads, the overhead may be worse.

---

References

1. Yuan, A., Shen, T., & Zhang, D. (2026). *Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoning*. arXiv:2605.09490 [cs.CL]. 2. Aminabadi, R. Y., et al. (2022). *DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Scale*. 3. Liu, Z., et al. (2024). *R-KV: Reducing KV Cache Through Importance-Based Eviction for LLM Long-Context Inference*.