[论文] FVAttn: Adaptive Sparse Attention with Runtime Load Balancing for Video Generation

论文概要

研究领域: cs.CV
作者: Hao Liu, Chenghuan Huang, Ye Huang
发布时间: 2026-07-21
arXiv: 2507.15490

中文摘要

视频扩散Transformer处理长时空序列，使得自注意力成为高分辨率视频生成的主要瓶颈。免训练稀疏注意力降低了这一开销，但自适应Top-p路由在多GPU序列并行下会产生不均匀的每头工作负载。由此产生的工作负载异质性将稀疏注意力转化为秩级掉队者问题。我们提出FVAttn——一种免训练的稀疏注意力系统，提升多GPU序列并行下自适应稀疏注意力的分布式执行效率。FVAttn使用Top-p路由、Top-k安全下限和视频感知块组织作为稀疏路由前端，然后在运行时修复物化掩码。运行时负载均衡通过P2P通信迁移少量重载头以缩短当前关键路径。松弛感知稀疏增强用额外的高价值块填充剩余的非关键秩松弛，而重叠机制将调度和迁移开销隐藏在现有计算之后。在步数蒸馏的Wan2.2 I2V上，FVAttn将平均负载不平衡从1.34降至1.08，相比FlashAttention实现4.41倍注意力加速，同时实现2.02-2.11倍DiT推理加速且视频质量具有竞争力。

原文摘要

Video Diffusion Transformers process long spatio-temporal sequences, making self-attention the main bottleneck in high-resolution video generation. Training-free sparse attention reduces this cost, but adaptive Top-p routing creates uneven per-head workloads under multi-GPU sequence parallelism. The resulting workload heterogeneity turns sparse attention into a rank-level straggler problem. We present FVAttn, a training-free sparse-attention system that improves the distributed execution efficiency of adaptive sparse attention under multi-GPU sequence parallelism. FVAttn uses Top-p routing, a Top-k safety floor, and video-aware block organization as the sparse-routing frontend, then repairs the materialized mask at runtime. Runtime Load Balancing migrates a small number of heavy heads via P2P communication to shorten the current critical path. Slack-Aware Sparse Augmentation fills residual non-critical-rank slack with additional high-value blocks, while overlap hides scheduling and migration overhead behind existing computation. On step-distilled Wan2.2 I2V, FVAttn reduces average load imbalance from 1.34 to 1.08 and delivers a 4.41x attention speedup over FlashAttention, while achieving a 2.02-2.11x DiT inference speedup with competitive video quality.

自动采集于 2026-07-21

#论文 #arXiv #CV #小凯