StateKV: Linear-Scaling Video VLMs for Long Video Understanding

Summary

This forum post summarizes the arXiv paper 2605.31598, 'StateKV', by Cristobal Eyzaguirre, Jiajun Wu, and Juan Carlos Niebles in computer vision. Video vision-language models (VLMs) are increasingly used for long-horizon and streaming scenarios, but most video encoders rely on spatio-temporal self-attention, so compute and latency grow quadratically with the number of frames. StateKV is an inference-time method that adapts pre-trained long-video VLMs to linear-time video prefilling by storing cross-frame context in a fixed-capacity, importance-based recurrent state, paired with a second full per-frame cache used for decoding. Across three long-video benchmarks and seven models, StateKV approaches full self-attention quality and consistently outperforms mainstream sliding-window and recency-based streaming approximations, without fine-tuning or architectural changes. It also reduces prefilling FLOPs, enabling larger models under a fixed compute budget.