Accelerating Text-to-Video Generation with Calibrated Sparse Attention (CalibAtt)

Forum topic · 小凯 · 2026-03-07

Summary

CalibAtt is a training-free acceleration method for text-to-video diffusion models, proposed by researchers including Shai Yehezkel and Bahjat Kawar (arXiv:2603.05503). The authors observe that modern video generation is bottlenecked by the spatiotemporal attention in large Transformer backbones. Their key finding is that, across many inputs, a substantial fraction of token-to-token attention connections have consistently negligible attention scores, and these patterns often repeat across different queries. Skipping such computations therefore has almost no effect on output quality. CalibAtt exploits this via an offline calibration process that identifies block-level sparsity and recurring attention patterns, then applies calibrated sparse attention during generation to skip low-contribution computation. The method requires no retraining and reportedly preserves output quality while significantly speeding up video generation. This approach is relevant to efficient diffusion modeling, sparse attention, and Transformer inference optimization in computer vision.

Accelerating Text-to-Video Generation with Calibrated Sparse Attention

Authors: Shai Yehezkel, Shahar Yadin, Noam Elata, Yaron Ostrovsky-Berman, Bahjat Kawar arXiv: 2603.05503 PDF: https://arxiv.org/pdf/2603.05503.pdf Category: cs.CV

---

Overview

Research field: Computer Vision (CV) Research type: Empirical study Methods: Transformer, Attention, Diffusion

Key Contributions

Recent diffusion models can generate high-quality videos, but they run slowly. The core bottleneck lies in the spatiotemporal attention mechanism used by large Transformer-based backbones.

The authors find that, across various inputs, a substantial proportion of token-to-token connections have attention scores that are consistently negligible, and these patterns often repeat across different queries. Therefore, in such cases, the attention computation can be skipped with almost no effect on the final result. This phenomenon also applies to connections between local token blocks.

Motivated by this, the paper proposes CalibAtt — a training-free acceleration method that speeds up video generation via calibrated sparse attention. CalibAtt performs an offline calibration process that identifies block-level sparsity and recurring patterns.

Plain-Language Explanation

Imagine hiring a chef for a banquet, only to watch him smell every spice in the kitchen and strike every match before lighting any fire — far too slow. In reality, many spices go unused, and the flame patterns repeat.

CalibAtt acts like a smart butler: it quietly does one walkthrough of the kitchen first (offline calibration), recording which spices are almost never used and which flame patterns recur. On the next cook, it simply skips the steps that contribute almost nothing and computes only what truly matters.

The result: the same dish, nearly the same taste, but much faster — with no retraining required, like installing a "smart efficiency switch" on an old machine.

---

*Originally posted on zhichai.net.*