Paper Overview
Research Area: Machine Learning
Authors: Haocheng Xi, Yiming Xie, Hexu Zhao, Yiwen Zhang, Michael Liu, Thomas Creavin, Kurt Keutzer, Xiuyu Li, Zhaoyang Lv, Chenfeng Xu, Haiwen Feng
Published: 2026-09-17
arXiv: 2609.20744
Key Points
- Problem: Video diffusion models repeatedly process very long spatiotemporal token sequences during denoising, making attention the dominant computational bottleneck.
- Motivation: Linear attention, widely adopted in recent LLMs, is an appealing alternative, but applying it directly to video models often fails to preserve the fine-grained interactions required for high-quality generation.
- Approach — Video DeltaNet (VDN): Combines local Softmax attention with bidirectional linear memory to carry long-range video context.
- Video Delta Attention (VDA): The linear branch updates memory once per frame by jointly incorporating the frame's spatial tokens.
- Branch calibration: Separate output projections and learnable gates calibrate the two branches; a staged teacher-alignment recipe progressively introduces the new pathway into a pretrained model.
- Instantiation: VDN is built on MiniMax H3. Hybrid attention applies to video-video interactions; interactions involving text or audio retain Softmax attention.
- Results: With eight-step distillation and an optimized SGLang serving stack, VDN-H3 completes DiT denoising of a 14.3-second, 768p video in only 6.70 seconds on eight NVIDIA B200 GPUs — a 14.5x speedup over a 50-step dense H3 baseline using the same number of GPUs.
- Paper: https://arxiv.org/abs/2609.20744
Links
*Auto-collected on 2026-09-20*