Video DeltaNet: A Video-Native Hybrid Attention for Livestream Video Generation

Forum topic · 小凯 · 2026-09-20

Summary

Video DeltaNet (VDN) is a video-native hybrid attention architecture for video diffusion models, presented in arXiv paper 2609.20744. Video diffusion models repeatedly process extremely long spatiotemporal token sequences during denoising, making attention the dominant computational bottleneck. While linear attention has been widely adopted in large language models, applying it directly to video models often fails to preserve fine-grained interactions needed for high-quality generation. VDN combines local Softmax attention with bidirectional linear memory for long-range video context. Its linear branch introduces Video Delta Attention (VDA), which updates memory once per frame by jointly incorporating spatial tokens, with separate output projections and learnable gates calibrating the two branches, plus a staged teacher-alignment recipe for progressively introducing new pathways into a pretrained model. Instantiated on MiniMax H3, VDN-H3 applies hybrid attention only to video-video interactions while keeping Softmax for text/audio interactions. Combined with eight-step distillation and an optimized SGLang serving stack, VDN-H3 denoises a 14.3-second 768p video in 6.70 seconds on eight NVIDIA B200 GPUs, a 14.5x speedup over a 50-step dense H3 baseline.

Paper Overview

Research Area: Machine Learning
Authors: Haocheng Xi, Yiming Xie, Hexu Zhao, Yiwen Zhang, Michael Liu, Thomas Creavin, Kurt Keutzer, Xiuyu Li, Zhaoyang Lv, Chenfeng Xu, Haiwen Feng
Published: 2026-09-17
arXiv: 2609.20744

Key Points

Video diffusion models repeatedly process long spatiotemporal token sequences during denoising, making attention the primary computational bottleneck.
Linear attention is an appealing alternative (widely adopted in recent LLMs), but applying it directly to video models often fails to preserve the fine-grained interactions required for high-quality generation.
Video DeltaNet (VDN) combines local Softmax attention with bidirectional linear memory to carry long-range video context.
The linear branch introduces Video Delta Attention (VDA): each frame updates the memory once by jointly incorporating its spatial tokens.
Separate output projections and learnable gates calibrate the two branches, and a staged teacher-alignment recipe progressively introduces the new pathways into a pretrained model.
VDN is instantiated on MiniMax H3, applying hybrid attention to video-video interactions, while interactions involving text or audio remain Softmax.

Performance

Combined with eight-step distillation and an optimized SGLang serving stack, VDN-H3 completes DiT denoising of a 14.3-second, 768p video in 6.70 seconds on eight NVIDIA B200 GPUs — a 14.5x speedup over a 50-step dense H3 baseline on the same number of cards.

Original Abstract (excerpt)

> Video diffusion models repeatedly process long spatiotemporal token sequences during denoising, making attention a major computational bottleneck. Linear attention offers an appealing alternative and has been widely adopted in recent large language models, but directly applying it to video models often fails to preserve the fine-grained interactions required for high-quality generation. We present Video DeltaNet (VDN), which combines local Softmax attention with bidirectional linear memory for long-range video context...

---

*Auto-collected on 2026-09-20*