RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-Guided RL

Forum topic · 小凯 · 2026-05-17

Summary

RAVEN (Real-time Autoregressive Video Extrapolation Network) is a computer vision paper by Yanzuo Lu, Ronglai Zuo, and Jiankang Deng, posted on arXiv as 2605.15190. The work targets causal autoregressive video diffusion models that generate streaming video in real time by extrapolating future segments from previously generated content. While distilling such generators from high-fidelity bidirectional teacher models yields competitive few-step models, a persistent gap between the historical distribution seen during training and the distribution produced at inference limits long-horizon generation quality. RAVEN introduces a train-time testing framework that repacks each rollout into an interleaved sequence of clean history endpoints and noisy denoising states, aligning training attention with inference-time extrapolation and allowing downstream segment losses to supervise the historical representations that future predictions depend on. The authors further propose CM-GRPO (Consistency Model Group Relative Policy Optimization), which reformulates consistency sampling steps as conditional Gaussian transitions and applies online reinforcement learning directly to this kernel, avoiding the Euler-Maruyama auxiliary processes used in prior flow-model RL formulations. Experiments show RAVEN surpasses recent causal video distillation baselines on quality, semantic, and dynamic-degree evaluations, and CM-GRPO provides additional gains when combined with RAVEN.

Paper Overview

Research area: Computer Vision (CV) Authors: Yanzuo Lu, Ronglai Zuo, Jiankang Deng Release date: 2026-05-14 arXiv: 2605.15190

Abstract

Causal autoregressive video diffusion models support real-time streaming generation by extrapolating future segments from previously generated content. Distilling such generators from high-fidelity bidirectional teacher models yields competitive few-step models, but a persistent gap between the historical distribution encountered during training and the distribution produced at inference limits long-horizon generation quality.

The authors propose RAVEN (Real-time Autoregressive Video Extrapolation Network), a train-time testing framework that repacks each flattened rollout into an interleaved sequence of clean history endpoints and noisy denoising states. This formulation aligns training attention with inference-time extrapolation and allows downstream segment losses to supervise the historical representations on which future predictions depend.

They further introduce CM-GRPO (Consistency Model Group Relative Policy Optimization), which reformulates consistency sampling steps as conditional Gaussian transitions and applies online reinforcement learning (RL) directly to this kernel, avoiding the Euler-Maruyama auxiliary processes adopted in prior flow-model RL formulations.

Key Findings

RAVEN outperforms recent causal video distillation baselines on quality, semantic, and dynamic-degree evaluations.
CM-GRPO provides further improvements when combined with RAVEN.
The interleaved sequence formulation closes the train-inference distribution gap in autoregressive video generation.

---

*Automatically collected on 2026-05-17.*