Summary
FVAttn is a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention for video diffusion transformers under multi-GPU sequence parallelism. While adaptive Top-p routing reduces self-attention cost, it creates uneven per-head workloads across GPUs, turning sparse attention into a rank-level straggler problem. FVAttn combines Top-p routing, a Top-k safety floor, and video-aware block organization as its sparse-routing frontend, then repairs the materialized mask at runtime. Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the critical path; Slack-Aware Sparse Augmentation fills non-critical-rank slack with additional high-value blocks; and an overlap mechanism hides scheduling and migration overhead behind existing computation. On step-distilled Wan2.2 I2V, FVAttn reduces average load imbalance from 1.34 to 1.08, achieves a 4.41x attention speedup over FlashAttention, and delivers a 2.02-2.11x end-to-end DiT inference speedup with competitive video quality. Paper: arXiv 2507.15490.
Paper Overview
- Field: cs.CV (computer vision / video generation)
- Authors: Hao Liu, Chenghuan Huang, Ye Huang
- Published: 2026-07-21
- arXiv: 2507.15490
Abstract
Video Diffusion Transformers process long spatio-temporal sequences, making self-attention the main bottleneck in high-resolution video generation. Training-free sparse attention reduces this cost, but adaptive Top-p routing creates uneven per-head workloads under multi-GPU sequence parallelism. The resulting workload heterogeneity turns sparse attention into a rank-level straggler problem.The authors present FVAttn, a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism. FVAttn uses Top-p routing, a Top-k safety floor, and video-aware block organization as the sparse-routing frontend, then repairs the materialized mask at runtime:
- Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the current critical path.
- Slack-Aware Sparse Augmentation fills residual non-critical-rank slack with additional high-value blocks.
- Overlap hides scheduling and migration overhead behind existing computation.
Results
On step-distilled Wan2.2 I2V:
- Average load imbalance reduced from 1.34 to 1.08
- 4.41x attention speedup over FlashAttention
- 2.02-2.11x DiT inference speedup with competitive video quality
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178446967