ReasonAlloc: Hierarchical Decoding-Time KV Cache Budget Allocation for LLM Reasoning

Forum topic · 小凯 · 2026-06-11

Summary

ReasonAlloc is a training-free framework that reformulates decoding-time KV cache compression in large language models as a hierarchical budget allocation problem. Long chain-of-thought (CoT) reasoning causes rapid KV cache growth and severe inference bottlenecks, but existing token-eviction methods typically assume a uniform budget across all layers and heads, while prior non-uniform allocation methods target static prompt prefilling rather than autoregressive reasoning. ReasonAlloc combines two complementary strategies: an offline layer-wise preallocation strategy that captures architecture-driven demand patterns (a 'reasoning wave'), and an online head-wise strategy that reallocates resources at decoding time based on real-time utility, directing budget to the most informative heads. Experiments on MATH-500 and AIME 2024 using DeepSeek-R1-Distill and AceReason models show the largest gains under tight KV budgets of 128-512 tokens. The paper is available on arXiv (2606.11164).

Paper Overview

Field: Machine Learning
Authors: Wenhao Liu, Hao Shi, Yunhe Li, Weizhi Fei, Xiangyuan Wang, Mengzhe Ruan, Hanxu Hou, Peisong Wang, Linqi Song, Shuang Qiu
Published: 2026-06-09
arXiv: 2606.11164

Abstract

Long chain-of-thought (CoT) trajectories in large language model (LLM) reasoning cause severe inference bottlenecks due to rapid key-value (KV) cache growth. Current decoding-time compression methods mitigate this issue via token eviction, but typically assume a uniform budget distribution across all layers and heads. In contrast, existing non-uniform budget allocation methods are predominantly designed for the static prompt prefill phase, and they do not capture the stepwise context demands of autoregressive reasoning.

To bridge this gap, the authors propose ReasonAlloc, a training-free framework that recasts decoding-time KV compression as a hierarchical budget allocation problem. ReasonAlloc operates at two complementary levels:

1. Offline layer-wise preallocation — captures architecture-driven demand patterns (a "reasoning wave") across layers. 2. Online head-wise reallocation — during decoding, reallocates budget in real time based on utility, favoring informative attention heads.

Results

Evaluated on MATH-500 and AIME 2024 using DeepSeek-R1-Distill and AceReason models, ReasonAlloc achieves its largest gains under tight KV cache budgets of 128-512 tokens.

---

*Auto-collected on 2026-06-11.*