RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-GRPO

Forum topic · 小凯 · 2026-05-16

Summary

RAVEN (Real-time Autoregressive Video Extrapolation Network) is a framework by Yanzuo Lu, Ronglai Zuo, and Jiankang Deng that improves real-time streaming video generation with causal autoregressive video diffusion models. While distilling high-fidelity bidirectional teacher models into few-step generators yields competitive results, a persistent gap between training-time and inference-time history distributions limits quality over long horizons. RAVEN addresses this with a training-time test framework that repacks each self rollout into an interleaved sequence of clean historical endpoints and noisy denoising states, aligning training attention with inference-time extrapolation and letting downstream chunk losses supervise history representations. The authors further propose Consistency Model Group Relative Policy Optimization (CM-GRPO), which reformulates consistency sampling steps as conditional Gaussian transitions and applies online reinforcement learning directly to this kernel, avoiding Euler-Maruyama auxiliary processes used in prior flow-model RL formulations. Experiments show RAVEN surpasses recent causal video distillation baselines on quality, semantic, and dynamics evaluations, with CM-GRPO providing additional gains. Paper: arXiv 2505.08629.

Overview

Field: Computer Vision (CV) Authors: Yanzuo Lu, Ronglai Zuo, Jiankang Deng arXiv: 2505.08629

Key Contributions

Causal autoregressive video diffusion models support real-time streaming generation by extrapolating future chunks from previously generated content. Distilling such generators from high-fidelity bidirectional teachers yields competitive few-step models, yet a persistent gap between the history distributions encountered during training and those arising at inference constrains generation quality over long horizons.

RAVEN (Real-time Autoregressive Video Extrapolation Network): a training-time test framework that repacks each self rollout into an interleaved sequence of clean historical endpoints and noisy denoising states. This formulation aligns training attention with inference-time extrapolation and allows downstream chunk losses to supervise the history representations that future predictions depend on.
CM-GRPO (Consistency Model Group Relative Policy Optimization): reformulates consistency sampling steps as conditional Gaussian transitions and applies online reinforcement learning directly to this kernel, avoiding the Euler-Maruyama auxiliary process adopted in previous flow-model RL formulations.

Results

Experiments show that RAVEN surpasses recent causal video distillation baselines on quality, semantic, and dynamics evaluations. Combining CM-GRPO with RAVEN brings further improvements.