Video-Mirai: Autoregressive Video Diffusion Models Need Foresight

Forum topic · 小凯 · 2026-06-04

Summary

Video-Mirai (arXiv 2606.03971) is a training-only method for streaming autoregressive video diffusion that addresses a representation-level planning gap: standard causal training only requires each causal state to explain the current segment, which can discard identity, layout, and motion information needed for a consistent future. The proposed approach keeps inference fully causal while introducing foresight during training: the generator rolls out causally, a frozen non-causal foresight encoder reads the completed rollout, and a lightweight predictor distills stopped-gradient targets from that encoder into the generator's causal states. Future frames supervise representations only, never generator inputs. At inference time the encoder and predictor are discarded, preserving the original architecture, per-step FLOPs, and KV-cache behavior. Authors: Yonghao Yu, Lang Huang, Runyi Li, Zerun Wang, Toshihiko Yamasaki.

Overview

Field: Computer Vision Authors: Yonghao Yu, Lang Huang, Runyi Li, Zerun Wang, Toshihiko Yamasaki arXiv: 2606.03971

Key Idea

Causal video generators must predict from the past, but they need not learn only from it. In streaming autoregressive video diffusion, each emitted segment becomes a commitment that future segments must preserve. Standard training, however, only asks each causal state to explain the present. This creates what the authors call a representation-level planning gap: states that fit the current segment may discard identity, layout, and motion information needed for a consistent future.

Method

Video-Mirai is a training-only method that closes this gap without changing causal inference:

1. The generator rolls out causally, as usual. 2. A frozen foresight encoder reads the completed rollout non-causally. 3. A lightweight predictor distills the resulting stopped-gradient targets into the generator's causal states.

Crucially, future frames supervise representations only — never the generator's inputs.

Inference

At inference time, the encoder and predictor are dropped entirely, preserving the original architecture, per-step FLOPs, and KV-cache behavior.

Paper Abstract (original)

> Causal video generators must predict from the past, but they need not learn only from it. In streaming autoregressive video diffusion, each emitted segment becomes a commitment that future segments must preserve. Standard training, however, only asks each causal state to explain the present. This creates what we call a representation-level planning gap: states that fit the current segment may discard identity, layout, and motion information needed for a consistent future. We introduce Video-Mirai, a training-only method that closes this gap without changing causal inference.

---

*Auto-collected on 2026-06-04.*