SeaCache: Spectral-Evolution-Aware Cache for Accelerating Diffusion Models (CVPR 2026 Best Paper Finalist)

Forum topic · 小凯 · 2026-06-19

Summary

SeaCache, a CVPR 2026 Oral and Best Paper Finalist from Sungkyunkwan University and NAVER Cloud, accelerates diffusion model inference with a spectral-evolution-aware caching strategy. Its key insight: diffusion denoising exhibits spectral evolution—early timesteps build low-frequency structure while later timesteps refine high-frequency detail—yet existing caching methods (DeepCache, TeaCache, TaylorSeer) measure feature distance in the raw feature space, where content and noise changes are entangled. SeaCache introduces the SEA Filter, derived from the optimal linear denoiser and a power-law prior on natural image spectra: features are transformed via FFT, multiplied by a timestep-dependent frequency response, and inverse-transformed back, so cache decisions track content changes rather than noise fluctuations. The filter adds only ~0.2% inference overhead, requires no training and no extra hyperparameters. On FLUX.1-dev, SeaCache reaches 2.22x-3.27x speedup, outperforming TeaCache by about 5.5 dB PSNR at similar latency. On HunyuanVideo, the gap widens to nearly 9 dB PSNR with 2x speedup, and similar gains hold on Wan2.1 1.3B. The approach is orthogonal to distillation and parallelization and is plug-and-play for AIGC pipelines. Paper: arXiv:2602.18993; code on GitHub.

Paper: SeaCache: Spectral-Evolution-Aware Cache for Accelerating Diffusion Models Authors: Jiwoo Chung, Sangeek Hyun, MinKyu Lee, Byeongju Han, Geonho Cha, Dongyoon Wee, Youngjun Hong, Jae-Pil Heo Venue: CVPR 2026 Oral, Best Paper Finalist Affiliations: Sungkyunkwan University + NAVER Cloud Paper: arXiv:2602.18993 Code: github.com/jiwoogit/SeaCache

The Problem: Why Diffusion Models Are Slow

Diffusion generation is inherently serial: each denoising step depends on the previous one, and every step requires a full forward pass. FLUX.1-dev at 50 steps means 50 full Transformer inferences—about 20 seconds per image even on high-end GPUs. Acceleration therefore follows two paths: (1) reducing step count via distillation, or (2) reducing per-step computation via quantization, pruning, efficient attention, and feature caching. SeaCache takes the second path.

The Blind Spot in Existing Caching

Caching exploits the fact that adjacent timesteps produce highly similar intermediate features (as in DeepCache, TeaCache, TaylorSeer). But diffusion denoising exhibits spectral evolution: early timesteps establish low-frequency structure (layout, contours), while later timesteps refine high-frequency detail (texture, edges, text).

Existing methods measure distance in the raw feature space, where content change and noise change are mixed—so cache decisions follow noise rather than content, caching at the wrong times and wasting compute or degrading quality.

The Core Idea: Decide in the Frequency Domain

SeaCache passes features through a Spectral-Evolution-Aware (SEA) Filter before measuring distance. The filter's response is derived from the optimal linear denoiser for $x_t = a_t x_0 + b_t \epsilon$ :

G_t(f) = \frac{a_t S_x(f)}{a_t^2 S_x(f) + b_t^2}

with natural-image power spectra $S_x(f) \propto f^{-\gamma}$ . Early timesteps (small $$a_t$$ , large $$b_t$$ ) suppress high frequencies; later timesteps progressively open them—exactly matching spectral evolution.

Implementation is lightweight: FFT → multiply by normalized $G_t^{norm}(f)$ → iFFT:

P(G_t, I_t) = \text{iFFT}\left(G_t^{norm}(f) \odot \text{FFT}(I_t)\right)

Cache scheduling then accumulates the filtered relative distance and refreshes the cache only when a threshold is exceeded. The filter costs only ~0.2% of total inference time, and SeaCache adds no extra hyperparameters—no retention ratios or coefficients to tune.

Results

FLUX.1-dev (text-to-image):

| Method | Latency(s) | Speedup | PSNR↑ | LPIPS↓ | SSIM↑ | |--------|-----------|---------|-------|--------|-------| | Original (50 steps) | 20.9 | 1.0x | — | — | — | | TeaCache (δ=0.3) | 11.4 | 1.83x | 20.76 | 0.211 | 0.810 | | SeaCache (δ=0.3) | 9.4 | 2.22x | 26.29 | 0.106 | 0.893 | | TeaCache (δ=0.6) | 7.1 | 2.94x | 17.21 | 0.348 | 0.714 | | SeaCache (δ=0.6) | 6.4 | 3.27x | 21.33 | 0.226 | 0.798 |

At comparable speedups, SeaCache beats TeaCache by 5.5 dB PSNR with roughly half the LPIPS.

HunyuanVideo (text-to-video):

| Method | Latency(s) | PSNR↑ | LPIPS↓ | SSIM↑ | |--------|-----------|-------|--------|-------| | TeaCache (δ=0.12) | 98.5 | 23.40 | 0.133 | 0.805 | | SeaCache (δ=0.19) | 90.8 | 32.39 | 0.047 | 0.932 | | TeaCache (δ=0.2) | 64.4 | 20.42 | 0.172 | 0.734 | | SeaCache (δ=0.35) | 58.1 | 26.46 | 0.133 | 0.857 |

At similar latency, the PSNR gap reaches nearly 9 dB; gains are also consistent on Wan2.1 1.3B. Qualitatively, SeaCache preserves in-image text (e.g., "QUANTUM" on a menu) and fine scene detail that baselines lose, with better temporal consistency in video.

Why It Works

1. Frequency-domain separation of signal and noise: cache decisions track content, not stochastic residuals. 2. Timestep-aware filtering: strictly low-pass early, progressively opening later—aligned with the coarse-to-fine generation rhythm. 3. Zero extra hyperparameters: training-free, plug-and-play.

Limitations and Future Directions

Benefits shrink for heavily distilled few-step models (e.g., FLUX.1-schnell), where adjacent-timestep differences are larger, though the spectral idea may transfer with filter redesign.
SeaCache is orthogonal to parallelization (e.g., DistriFusion) and efficient attention; gains can stack.
Future work: learnable or content-adaptive filter responses, and end-to-end optimization.
Video generation benefits most, likely because the temporal dimension amplifies the content-vs-noise separation gain.

Takeaways

For AIGC product teams: drop-in 2–3x inference speedup with minimal quality loss, no retraining or architecture changes—especially valuable for video generation.
For researchers: the frequency domain is more informative than the raw feature space for analyzing and scheduling diffusion denoising.
For hardware vendors: the FFT → pointwise multiply → iFFT pattern maps well to fixed-function acceleration units.

In short, SeaCache shifts cache scheduling from "following noise" to "following content," at essentially zero cost—a simple, theory-grounded observation that explains its Best Paper Finalist recognition.

References

Chung, J., et al. SeaCache: Spectral-Evolution-Aware Cache for Accelerating Diffusion Models. *CVPR* (2026). arXiv:2602.18993
Ma, X., Fang, G., & Wang, X. DeepCache: Accelerating Diffusion Models for Free. *CVPR* (2024).
Chen, X., et al. TeaCache: Temporal Feature Cache for Video Diffusion Model. *NeurIPS* (2025).
Liu, Y., et al. TaylorSeer: Taylor-Series Expansion for Accelerating Diffusion Models. *NeurIPS* (2025).
Kahatapitiya, K., et al. AdaCache: Adaptive Cache for Video Diffusion Models. *CVPR* (2024).