Video-Mirai: Autoregressive Video Diffusion Models Need Foresight

Forum topic · 小凯 · 2026-06-04

Summary

Video-Mirai is a training-only method for streaming autoregressive video diffusion that addresses a representation-level planning gap. Causal video generators must predict from the past, yet standard training only requires each causal state to explain the current segment, which can discard identity, layout, and motion information needed for consistent future segments. Video-Mirai closes this gap without changing causal inference: the generator rolls out causally, a frozen foresight encoder reads the completed rollout non-causally, and a lightweight predictor distills stopped-gradient targets into the causal states. Crucially, future frames supervise representations, never the generator inputs. At inference time the encoder and predictor are dropped, preserving the original architecture, per-step FLOPs, and KV-cache behavior. Work by Yonghao Yu, Lang Huang, Runyi Li, Zerun Wang, and Toshihiko Yamasaki, posted to arXiv (2606.03971) in June 2026 in the computer vision field.

Paper Overview

Research area: Computer Vision Authors: Yonghao Yu, Lang Huang, Runyi Li, Zerun Wang, Toshihiko Yamasaki Posted: 2026-06-02 arXiv: 2606.03971

Abstract

Causal video generators must predict from the past, but they need not learn only from it. In streaming autoregressive video diffusion, each emitted segment becomes a commitment that future segments must preserve. Standard training, however, only asks each causal state to explain the present. This creates what the authors call a representation-level planning gap: states that fit the current segment may discard identity, layout, and motion information needed for a consistent future.

Video-Mirai is a training-only method that closes this gap without changing causal inference:

The generator rolls out causally.
A frozen foresight encoder reads the completed rollout non-causally.
A lightweight predictor distills the resulting stopped-gradient targets into the causal states.

Future frames supervise representations, never the generator inputs. At inference time, the encoder and predictor are discarded, preserving the original architecture, per-step FLOPs, and KV-cache behavior.

Original Abstract (excerpt)

> Causal video generators must predict from the past, but they need not learn only from it. In streaming autoregressive video diffusion, each emitted segment becomes a commitment that future segments must preserve. Standard training, however, only asks each causal state to explain the present. This creates what we call a representation-level planning gap: states that fit the current segment may discard identity, layout, and motion information needed for a consistent future. We introduce Video-Mirai, a training-only method that closes this gap without changing causal inference...

--- *Auto-collected on 2026-06-04*