BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing (arXiv 2610.31633)

Forum topic · 小凯 · 2026-10-25

Summary

Looped transformer architectures scale model quality while keeping parameter count fixed by repeatedly passing activations through the same block of layers. However, during inference each recurrence generates an independent KV-cache, causing memory usage to grow linearly with the number of loops. This paper targets KV-shareability: enabling the KV-cache of the final recurrence to be shared with earlier recurrences without quality loss. The authors propose a two-stage method. First, stochastic depth is used during pretraining to induce contractive residual dynamics that drive representations toward a common limit across recurrences. Second, a post-training method called BeNice! broadcasts the final recurrence's KV-cache into the first recurrence and trains the model to produce compatible residuals. BeNice! composes with various KV-compression techniques; in simulated low-memory long-context inference, it improves cache hit rate by 31.4%, throughput by 23.6%, and reduces p90 time-to-first-token by 83.6%.

Paper Overview

Field: Machine Learning Authors: Hayden Prairie, Jacob M. Ryan, Taylor Berg-Kirkpatrick, Daniel Y. Fu Published: 2026-10-22 arXiv: 2610.31633

Summary

Looped architectures scale quality while keeping parameter count fixed by repeatedly passing activations through the same block of layers. During inference, each recurrence generates independent KV-caches, causing memory to grow linearly. To address this challenge, prior work has proposed KV-sharing, finding that, in some cases, the KV-cache of the final recurrence can be shared with the earlier recurrences without quality loss — a property the authors denote as KV-shareability.

Method

The paper proposes a two-stage approach:

1. Pretraining with stochastic depth: This induces contractive residual dynamics that drive representations toward a common limit across recurrences. 2. BeNice! (post-training): The final recurrence's KV-cache is broadcast into the first recurrence, and the model is trained to produce compatible residuals.

Results

BeNice! makes KV-sharing composable with a variety of KV-compression methods. In simulated low-memory long-context inference:

Cache hit rate improved by 31.4%
Throughput improved by 23.6%
p90 time-to-first-token reduced by 83.6%

---

*Auto-collected on 2026-10-26.*

BeNice! Closing Looping's Inference-Memory Gap via KV-Sharing (arXiv 2610.31633)

Summary

Paper Overview

Summary

Method

Results

Tags