Mask Forcing: Improving Autoregressive Video Diffusion Distillation via Dual-Noise Masking Rollout

Forum topic · 小凯 · 2026-09-10

Summary

Autoregressive (AR) video diffusion models are promising for real-time video generation, and recent methods distill pretrained bidirectional video diffusion models into causal AR students using Distribution Matching Distillation (DMD). However, the distilled videos often suffer from over-saturation and over-smoothing, caused by the mode-seeking behavior of the reverse KL objective, which makes the student distribution collapse onto only a few modes of the teacher distribution. This paper proposes Mask Forcing, a Dual-Noise Masking Rollout strategy that perturbs the AR student's self-rollout to mitigate reverse-KL-induced mode collapse. Random masks inject cleaner signals along spatial and temporal axes during rollout, encouraging the student to explore more regions of the teacher distribution so DMD provides learning signals beyond already-covered modes. Cleaner tokens also act as denoising guidance for noisier tokens, improving intermediate predictions and reducing error accumulation. Experiments show the method improves multiple AR video diffusion distillation approaches with higher visual quality, without real video data or extra post-training stages. Paper: arXiv 2609.09123.

Overview

Field: cs.CV
Authors: Zhuoran Zhao, Shengju Qian, Tongtong Liang, Xianghao Kong, Songchun Zhang, Junchao Huang, Guian Fang, Xin Wang, Pan Hui, Anyi Rao
arXiv: 2609.09123

Abstract

To address this, the authors propose Mask Forcing, a Dual-Noise Masking Rollout strategy that perturbs the AR student self-rollout to mitigate mode collapse induced by reverse-KL mode seeking. The core idea is to inject cleaner signals into noisy rollout inputs via random masks along spatial and temporal axes during the self-rollout process of AR diffusion distillation. Such perturbations encourage the student rollouts to explore more regions of the teacher distribution, allowing DMD to provide learning signals beyond the modes already covered by the student. Moreover, the cleaner tokens act as denoising guidance for other noisier tokens, improving the intermediate rollout predictions and reducing error accumulation.

Extensive experiments demonstrate that this method improves multiple AR video diffusion distillation methods with higher visual quality efficiently, without incorporating real video data or additional post-training stages.

Key points

Problem: DMD-based AR video diffusion distillation suffers from reverse-KL mode-seeking, causing student mode collapse, over-saturation, and over-smoothing.
Method: Dual-Noise Masking Rollout — random spatial/temporal masking injects cleaner signals into noisy rollout inputs during self-rollout distillation.
Benefit 1: Students explore more of the teacher distribution, so DMD learning signals go beyond already-covered modes.
Benefit 2: Cleaner tokens guide denoising of noisier tokens, improving intermediate rollout predictions and reducing error accumulation.
Result: Consistent visual-quality improvements across multiple AR distillation methods, with no real video data or extra post-training required.