RAVEN: Real-time Autoregressive Video Extrapolation with Consistency-Model RL

Forum topic · 小凯 · 2026-05-16

Summary

RAVEN (Real-time Autoregressive Video Extrapolation Network) is a new framework for causal autoregressive video diffusion models that support real-time streaming generation by extrapolating future chunks from previously generated content. Distilling high-fidelity bidirectional teacher models yields competitive few-step generators, but a persistent gap between the history distributions seen during training and those arising at inference limits long-horizon quality. RAVEN introduces a training-time test framework that repacks each self rollout into an interleaved sequence of clean historical endpoints and noisy denoising states, aligning training attention with inference-time extrapolation and letting downstream chunk losses supervise history representations. The authors also propose CM-GRPO (Consistency Model Group Relative Policy Optimization), which reformulates consistency sampling steps as conditional Gaussian transitions and applies online reinforcement learning directly to this kernel, avoiding the Euler-Maruyama auxiliary processes used in prior flow-model RL formulations. Experiments show RAVEN surpasses recent causal video distillation baselines on quality, semantic, and dynamic-degree evaluations, with further gains when combined with CM-GRPO. Paper: arXiv:2505.08629.

Paper Overview

Field: Computer Vision (CV)
Authors: Yanzuo Lu, Ronglai Zuo, Jiankang Deng
Release date: 2026-05-16
arXiv: 2505.08629

Abstract

Causal autoregressive video diffusion models support real-time streaming generation by extrapolating future chunks from previously generated content. Distilling such generators from high-fidelity bidirectional teachers yields competitive few-step models, yet a persistent gap between the history distributions encountered during training and those arising at inference constrains generation quality over long horizons. We introduce the Real-time Autoregressive Video Extrapolation Network (RAVEN), a training-time test framework that repacks each self rollout into an interleaved sequence of clean historical endpoints and noisy denoising states. This formulation aligns training attention with inference-time extrapolation and allows downstream chunk losses to supervise the history representations that future predictions rely on. We further propose Consistency Model Group Relative Policy Optimization (CM-GRPO), which reformulates consistency sampling steps as conditional Gaussian transitions and applies online reinforcement learning (RL) directly to this kernel, avoiding the Euler-Maruyama auxiliary processes adopted in prior flow-model RL formulations. Experiments show that RAVEN surpasses recent causal video distillation baselines across quality, semantic, and dynamic-degree evaluations, and CM-GRPO yields further improvement when combined with RAVEN.

Key Contributions

1. RAVEN training-time test framework: repacks each autoregressive self rollout into an interleaved sequence of clean historical endpoints and noisy denoising states, closing the train–inference history distribution gap. 2. Aligned supervision: downstream chunk losses can supervise the history representations on which future predictions depend. 3. CM-GRPO: treats consistency sampling steps as conditional Gaussian transitions and applies online RL directly to this kernel, without Euler-Maruyama auxiliary processes used in earlier flow-model RL methods.

Results

RAVEN outperforms recent causal video distillation baselines on quality, semantic, and dynamic-degree metrics; adding CM-GRPO brings further gains.

--- *Auto-collected on 2026-05-16*