Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout

Forum topic · 小凯 · 2026-09-10

Summary

Mask Forcing is a new strategy for improving autoregressive (AR) video diffusion distillation, proposed by Zhuoran Zhao, Shengju Qian, and colleagues at HKUST (paper: arXiv 2609.09123). AR video diffusion models show strong potential for real-time video generation, and recent approaches distill pretrained bidirectional video diffusion models into causal AR students via Distribution Matching Distillation (DMD). However, generated videos often suffer from over-saturation and over-smoothing, caused by the mode-seeking behavior of the reverse KL objective, which collapses the student distribution onto only a few modes of the teacher distribution. Mask Forcing introduces a Dual-Noise Masking Rollout that perturbs the AR student's self-rollout by injecting cleaner signals into noisy rollout inputs through random masks along spatial and temporal axes. This encourages the student to explore more regions of the teacher distribution, so DMD delivers learning signals beyond modes already covered. Cleaner tokens also act as denoising guidance for noisier tokens, improving intermediate rollout predictions and reducing error accumulation. Experiments show the method improves multiple AR video diffusion distillation approaches with higher visual quality, without requiring real video data or extra post-training stages.

Paper Overview

Field: cs.CV Authors: Zhuoran Zhao, Shengju Qian, Tongtong Liang, Xianghao Kong, Songchun Zhang, Junchao Huang, Guian Fang, Xin Wang, Pan Hui, Anyi Rao arXiv: 2609.09123

Abstract

Autoregressive (AR) video diffusion models have shown great potential in real-time video generation. Recent methods distill pretrained bidirectional video diffusion models into causal AR students through Distribution Matching Distillation (DMD), but the generated videos often suffer from over-saturation and over-smoothing issues, resulting in limited visual quality and realism. The key contributing factor is the mode-seeking behavior of the reverse KL objective in DMD, which can cause the student distribution to collapse onto only a few modes of the teacher distribution.

To address this, the authors propose Mask Forcing, a Dual-Noise Masking Rollout strategy that perturbs the AR student self-rollout to mitigate mode collapse induced by reverse-KL mode seeking. The core idea is to inject cleaner signals into noisy rollout inputs via random masks along spatial and temporal axes during the self-rollout process of AR diffusion distillation. Such perturbations encourage the student rollouts to explore more regions of the teacher distribution, allowing DMD to provide learning signals beyond the modes already covered by the student. Moreover, the cleaner tokens act as denoising guidance for other noisier tokens, improving the intermediate rollout predictions and reducing error accumulation.

Key Findings

Extensive experiments demonstrate that Mask Forcing improves multiple AR video diffusion distillation methods with higher visual quality.
The improvement is achieved efficiently, without incorporating real video data or additional post-training stages.