AdaState: Self-Evolving Anchors for Streaming Video Generation

Forum topic · 小凯 · 2026-06-01

Summary

AdaState is a paper by Yusuf Dalva and Pinar Yanardag (arXiv 2605.30349) addressing a key limitation of autogeneous video diffusion models used for streaming video generation. These models generate clips frame by frame, conditioning each segment on previously generated content, and are structurally anchored to the first frame: its key-value representations hold a privileged position in the attention cache and serve as the primary scene reference throughout generation. Because this anchor is the cleanest, least error-prone entry in the cache, it attracts disproportionate attention, suppressing video dynamics, locking scene composition to the initial viewpoint, and producing temporally shallow videos where motion, camera movement, and scene progression are stifled by static consistency. AdaState replaces the static anchor with an adaptive state — a hidden latent variable processed in parallel during each clip's denoising but never rendered. Instead of referencing a frozen first frame, the model generates its own scene anchor at each step by attending to the previous state and current content, yielding a reference that evolves with the generated content. The formulation treats time as relative, introducing recursion into generation where denoising acts as the transition function and the KV cache as the carrier, with no external modules. Experiments show significantly improved video dynamics, richer motion, and natural scene progression.

Paper Overview

Research Area: Computer Vision (CV) Authors: Yusuf Dalva, Pinar Yanardag Published: 2026-05-28 arXiv: 2605.30349

Abstract

Autoregressive video diffusion models enable streaming video by generating clips conditioned on previously generated content. These models are structurally anchored to the first frame: its key-value representations occupy a privileged position in the attention cache and serve as the primary scene reference throughout generation.

As the cleanest, least error-prone position in the cache, this anchor attracts disproportionate attention, suppressing video dynamics and locking scene composition to the initial viewpoint — even as the scene naturally evolves. The result is temporally shallow video in which motion, camera movement, and scene progression are suppressed by static consistency.

The AdaState Approach

To address this, the authors replace the static anchor with an adaptive state — a hidden latent variable that the model processes in parallel during each clip's denoising but never renders. Instead of referencing a frozen first frame, the model generates its own scene anchor at each step by attending to the previous state and the current content, producing a reference that evolves alongside the generated content.

Unlike standard video generation, which encodes absolute notions of time, this formulation treats time as relative:

Every generation step sees the same pose structure, regardless of how far generation has progressed.
State transitions are identical for every clip.

Together, these properties introduce recursion into the generation process, where denoising acts as the transition function and the KV cache serves as the carrier — without requiring any external modules.

Results

Experiments demonstrate that the adaptive state significantly improves video dynamics, enabling richer motion and natural scene progression.

--- *Automatically collected on 2026-06-01*