StateKV: Linear-Scaling Video VLMs for Long Video Understanding

Summary

Video vision-language models (VLMs) are increasingly used for long-horizon and streaming applications, but most video encoders rely on spatiotemporal self-attention, causing compute and latency to grow quadratically with the number of frames. This paper introduces StateKV, a training-free, inference-time method that adapts pretrained long-video VLMs to linear-time video prefilling. StateKV maintains cross-frame context in a fixed-capacity, importance-based recurrent state, paired with a second full per-frame cache used for decoding. Across three long-video benchmarks and seven models, StateKV closely matches full self-attention and consistently outperforms mainstream sliding-window and recency-based streaming approximations, without fine-tuning or architectural changes. It also reduces FLOPs for video prefilling, enabling larger models to run within a fixed compute budget. Authors: Cristobal Eyzaguirre, Jiajun Wu, Juan Carlos Niebles. Preprint: arXiv 2605.31598.