KV-Fold: One-Step KV-Cache Recurrence for Training-Free Long-Context Inference

Forum topic · 小凯 · 2026-05-14

Summary

KV-Fold (arXiv:2605.12471) is a simple, training-free long-context inference protocol that treats the key-value (KV) cache as the accumulator in a left fold over sequence chunks, analogous to foldl in functional programming. At each step, the model processes the next chunk conditioned on the accumulated cache, appends newly produced keys and values, and passes the enlarged cache forward, reusing internal state across segments without modifying or retraining the model. Building on a KV cache concatenation primitive originally introduced for latent multi-agent communication, the authors repurpose it as a chunk-to-chunk recurrence. Despite its simplicity, the induced recurrence is stable: per-step drift saturates into a flat plateau that is insensitive to 10,000x numerical precision variations, robust across chunk sizes, and consistent across model families. KV-Fold preserves exact information over long distances, achieving 100% exact-match retrieval on needle-in-a-haystack benchmarks across 152 trials spanning 16K to 128K token contexts and chain depths up to 511 on Llama-3.1-8B, all within the memory limits of a single 40GB GPU.

Overview

Field: NLP Authors: Alireza Nadali, Patrick Cooper, Ashutosh Trivedi, Alvaro Velasquez Published: 2026-05-12 arXiv: 2605.12471

Abstract

We introduce KV-Fold, a simple, training-free long-context inference protocol that treats the key-value (KV) cache as the accumulator in a left fold over sequence chunks. At each step, the model processes the next chunk conditioned on the accumulated cache, appends the newly produced keys and values, and passes the enlarged cache forward; the same one-step update is applied repeatedly, analogous to foldl in functional programming.

Building on the KV cache concatenation primitive introduced for latent multi-agent communication, the authors repurpose it as a chunk-to-chunk recurrence for long-context inference. When processing chunk *t*, the model attends to the KV cache carried from earlier chunks as a prefix, reusing its internal state across segments without modifying or retraining the model.

Key Findings

Despite its simplicity, the induced recurrence is stable: per-step drift briefly rises and then saturates into a flat plateau that persists across deep chains.
This plateau is insensitive to 10,000x variations in numerical precision, robust across chunk sizes, and consistent across model families.
On needle-in-a-haystack benchmarks, KV-Fold preserves exact information over long distances: 100% exact-match retrieval across 152 trials, spanning contexts from 16K to 128K tokens and chain depths up to 511 on Llama-3.1-8B.
All results fit within the memory limits of a single 40GB GPU.

--- *Auto-collected on 2026-05-14*