KV-streams for Efficient Compaction in Agentic Reinforcement Learning

Forum topic · 小凯 · 2026-09-30

Summary

Scaling agentic LLMs is bottlenecked by the need to fit ever-growing context traces into GPU memory. Context compaction is the most popular mitigation, keeping GPU memory constant for a given trace, but most compaction strategies repeatedly re-prefill the LLM context, hurting training throughput. The authors propose KV-streams, a plug-and-play strategy compatible with any compaction method that substantially increases throughput with no evidence of performance degradation. KV-streams achieve scalable compaction by streaming the KV cache forward through the model rather than flushing it after each compaction. They demonstrate compatibility with three different compaction strategies, achieving training speedups of 2.6x to 5x. Beyond efficiency, streamed KV caches can act as a recurrent state that carries information no longer present in the context; in controlled environments, the authors show—unlike prior work—that RL alone suffices for this behavior to emerge. Paper: arXiv:2609.35750.

Paper Overview

Field: ML Authors: Emiliano Penaloza, Dane Malenfant, Dheeraj Vattikonda, Roger Creus Castanyer, Siddarth Venkatraman, Abhay Puri, Jonathan Light, Matthew James Sargent, Augustine N. Mavor-Parker, Massimo Caccia, Lucas Caccia, Glen Berseth, Esmeralda S. Whitammer, Alessandro Sordoni, Minseon Kim, Marc-Alexandre Côté, Laurent Charlin, Guillaume Lajoie Published: 2026-09-28 arXiv: 2609.35750

Summary (translated from Chinese forum post)

Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput.

To alleviate this bottleneck and enable efficient trainable compaction, the authors propose KV-streams — a plug-and-play strategy compatible with any compaction strategy that substantially increases throughput while showing no evidence of hindering performance. KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction.

Key results:

KV-streams are compatible with three different compaction strategies, achieving 2.6x to 5x training speedups.
Streamed KV caches can act as a recurrent state, carrying information that has long since disappeared from the context.
In controlled environments, unlike prior work, RL alone is shown to be sufficient for this behavior to emerge.

Overall, KV-streams are an efficient, lightweight plug-and-play add-on suitable for any post-training pipeline.

Original Abstract (excerpt)

> Scaling the horizon of agentic LLMs is bottlenecked by the need to fit ever longer context traces in GPU memory. Context compaction has been the most popular mechanism to alleviate this issue, keeping GPU memory constant for a given trace. Unfortunately, most compaction strategies rely on prefilling the LLM context many times over, hindering training throughput. To alleviate this bottleneck and enable efficient trainable compaction, we propose KV-streams, a plug-and-play strategy compatible with any compaction strategy that substantially increases throughput while showing no evidence of hindering performance. KV-streams enable scalable compaction by streaming the KV cache forward rather than flushing it after each compaction. We show that KV-streams enable three different compaction strate...

--- *Auto-collected on 2026-09-30.*