Overview
Field: cs.CV Authors: Hao Liu, Chenghuan Huang, Ye Huang Published: 2026-07-21 arXiv: 2507.15490
Abstract
Video Diffusion Transformers process long spatio-temporal sequences, making self-attention the main bottleneck in high-resolution video generation. Training-free sparse attention reduces this cost, but adaptive Top-p routing creates uneven per-head workloads under multi-GPU sequence parallelism. The resulting workload heterogeneity turns sparse attention into a rank-level straggler problem.
We present FVAttn, a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism. FVAttn uses Top-p routing, a Top-k safety floor, and video-aware block organization as the sparse-routing frontend, then repairs the materialized mask at runtime. Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the current critical path. Slack-Aware Sparse Augmentation fills residual non-critical-rank slack with additional high-value blocks, while overlap hides scheduling and migration overhead behind existing computation.
On step-distilled Wan2.2 I2V, FVAttn reduces average load imbalance from 1.34 to 1.08 and delivers a 4.41x attention speedup over FlashAttention, while achieving a 2.02-2.11x DiT inference speedup with competitive video quality.
--- *Auto-collected on 2026-07-21*