Do All Reasoning Tokens Need to Live in HBM? Semantics-Aware Memory Tiering for LLM KV Caches

Forum topic · 小凯 · 2026-05-18

Summary

Reasoning LLMs generate thousands of chain-of-thought tokens whose KV caches must typically reside in scarce GPU HBM. Common token eviction approaches fail badly for reasoning tasks—removing half the cache drops accuracy to zero. A paper by Yuan, Shen, and Zhang (arXiv:2605.09490) proposes instead a semantics-aware memory hierarchy that splits tokens into four tiers: HBM, DDR (CPU memory), compressed, and evicted. Low-importance tokens are not destroyed but offloaded to CPU memory and prefetched at full precision before each attention computation, appearing as if they never left the GPU. The core finding: accuracy depends only on how many tokens are permanently discarded (eviction ratio), not on how many remain in HBM. Across 7B–32B models and four benchmarks, evicting just 3% of tokens retains 91% of full-cache accuracy, and on a 14B model the method matches the uncompressed baseline (90% vs 86%) while halving HBM usage. Open questions remain around prefetch latency: the reported 5–7% transfer overhead may only hold with sufficient GPU–CPU bandwidth, and multi-user concurrent serving could degrade performance further.

Reasoning LLMs generate thousands of chain-of-thought tokens, and every token's KV cache must be stored in precious GPU HBM. The mainstream approach is to evict unimportant tokens from the cache—but this is catastrophic for reasoning tasks: removing half the cache drives accuracy to zero.

Yuan, Shen, and Zhang (arXiv:2605.09490) asked a different question: does every token have to live in HBM? Could some tokens reside somewhere else?

How it works

Their semantics-aware memory tiering scheme classifies tokens into four levels:

1. HBM — hot, high-importance tokens stay on the GPU 2. DDR — low-importance tokens are offloaded to CPU memory rather than destroyed 3. Compressed — tokens kept in a reduced representation 4. Evicted — tokens permanently discarded

Before each attention computation, offloaded tokens are prefetched back at full precision, as if they had never left the GPU.

Key findings

Accuracy depends only on how many tokens are permanently discarded (the eviction ratio), not on how many tokens remain in HBM.
Across 7B–32B models and four benchmarks, evicting only 3% of tokens retains 91% of full-cache accuracy.
On a 14B model, the method matches the uncompressed baseline (90% vs 86%) while halving HBM usage.

Open questions

The impact of prefetch latency is unclear—fetching tokens from DDR takes extra time. The paper reports transfer overhead of only 5–7%, but this likely holds only when GPU–CPU bandwidth is sufficient. Performance may be worse under multi-user concurrent serving.

References

1. Yuan, A., Shen, T., & Zhang, D. (2026). *Not All Thoughts Need HBM: Semantics-Aware Memory Hierarchy for LLM Reasoning*. arXiv:2605.09490 [cs.CL]. 2. Aminabadi, R. Y., et al. (2022). *DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Scale*. 3. Liu, Z., et al. (2024). *R-KV: Reducing KV Cache Through Importance-Based Eviction for LLM Long-Context Inference*.