ReasonAlloc: Hierarchical Decoding-Time KV Cache Budget Allocation for LLM Reasoning

Forum topic · 小凯 · 2026-06-11

Summary

ReasonAlloc is a training-free framework that reformulates decoding-time KV cache compression in large language models as a hierarchical budget allocation problem. Long chain-of-thought (CoT) reasoning causes rapid KV cache growth and severe inference bottlenecks. Existing token-eviction methods typically assume a uniform budget across layers and heads, while prior non-uniform allocation approaches target only the static prefill phase and miss the stepwise context demands of autoregressive reasoning. ReasonAlloc addresses this at two complementary levels: an offline layer-wise preallocation strategy captures architecture-driven demand patterns (called the "reasoning wave"), and an online head-wise strategy reallocates budget at decoding time to informative heads based on real-time utility. Experiments on MATH-500 and AIME 2024 with DeepSeek-R1-Distill and AceReason models show the largest gains under small budgets of 128-512 tokens. The paper (arXiv: 2606.11164) is authored by Wenhao Liu and colleagues.

Paper Overview

Field: Machine Learning (ML)
Authors: Wenhao Liu, Hao Shi, Yunhe Li, Weizhi Fei, Xiangyuan Wang, Mengzhe Ruan, Hanxu Hou, Peisong Wang, Linqi Song, Shuang Qiu
Published: 2026-06-09
arXiv: 2606.11164

Abstract

Long chain-of-thought (CoT) trajectories in large language model (LLM) reasoning cause severe inference bottlenecks due to rapid key-value (KV) cache growth. Current decoding-time compression methods mitigate this issue via token eviction, but typically assume a uniform budget distribution across all layers and heads. In contrast, existing non-uniform budget allocation methods are predominantly designed for the static prompt prefill phase, and they do not capture the stepwise context demands of autoregressive reasoning.

To bridge this gap, the authors propose ReasonAlloc, a training-free framework that recasts decoding-time KV compression as a hierarchical budget allocation problem. ReasonAlloc operates at two complementary levels:

Offline layer-wise preallocation: captures an architecture-driven demand pattern (the "reasoning wave") to preallocate budget across layers.
Online head-wise reallocation: during decoding, redistributes resources to informative heads based on real-time utility.

Results

Evaluated on MATH-500 and AIME 2024 using DeepSeek-R1-Distill and AceReason models, ReasonAlloc achieves its largest gains under small KV cache budgets (128-512 tokens).

---

*Auto-collected on 2026-06-11 from zhichai.net.*