SeaCache: Spectral-Evolution-Aware Caching for Accelerating Diffusion Models (CVPR 2026 Best Paper Finalist)

Forum topic · 小凯 · 2026-06-19

Summary

SeaCache, a CVPR 2026 Oral and Best Paper Finalist from Sungkyunkwan University and NAVER Cloud, accelerates diffusion model inference with a spectral-evolution-aware caching strategy. Unlike prior feature caching methods (DeepCache, TeaCache, TaylorSeer) that measure step-to-step differences in raw feature space—mixing content changes with noise—SeaCache applies a theoretically derived spectral evolution-aware (SEA) filter before measuring cache distance. Derived from the optimal linear denoiser and a power-law image spectrum prior, the timestep-dependent filter suppresses high frequencies early in sampling and gradually opens them later, matching how diffusion models build low-frequency structure first and refine high-frequency detail afterwards. The implementation is lightweight FFT → filtering → iFFT, adding only ~0.2% inference overhead, with no extra hyperparameters and zero training. On FLUX.1-dev, SeaCache reaches 2.22x-3.27x speedup with up to +5.5 dB PSNR over TeaCache; on HunyuanVideo, it delivers nearly +9 dB PSNR at comparable latency (~2x speedup), with similar gains on Wan2.1. The method is training-free, plug-and-play, and orthogonal to distillation and multi-GPU parallelism, making it readily deployable for image and video generation products.

Paper: SeaCache: Spectral-Evolution-Aware Cache for Accelerating Diffusion Models Authors: Jiwoo Chung, Sangeek Hyun, MinKyu Lee, Byeongju Han, Geonho Cha, Dongyoon Wee, Youngjun Hong, Jae-Pil Heo Venue: CVPR 2026 Oral, Best Paper Finalist Affiliations: Sungkyunkwan University + NAVER Cloud Paper: arXiv:2602.18993 Code: github.com/jiwoogit/SeaCache

Key points

Problem: Diffusion inference is inherently sequential (tens of full forward passes per image), so acceleration relies on reducing step count (distillation) or reducing per-step cost (quantization, caching). SeaCache targets the caching route.
Blind spot in prior caching: Existing methods (DeepCache, TeaCache, TaylorSeer) decide when to reuse features based on distances in raw feature space, which conflates content change with noise fluctuation—leading to wrong cache decisions in both directions.
Core insight — spectral evolution: Early timesteps establish low-frequency structure (layout, contours); later timesteps refine high-frequency detail (texture, edges, text). Diffusion generation is effectively a low-to-high-frequency spectral unfolding.
SEA Filter: Derived from the optimal linear denoiser with a power-law spectrum prior $S_x(f) \propto f^{-\gamma}$ :

G_t(f) = \frac{a_t S_x(f)}{a_t^2 S_x(f) + b_t^2}

Applied as FFT → pointwise multiply by normalized $G_t^{norm}(f)$ → iFFT:

P(G_t, I_t) = \text{iFFT}\left(G_t^{norm}(f) \odot \text{FFT}(I_t)\right)

Cost: ~0.2% of total inference time.

Dynamic scheduling: Cache refresh is triggered when accumulated filtered-feature relative distance exceeds a threshold, so decisions follow content rather than noise. No extra hyperparameters beyond the threshold; fully training-free and plug-and-play.

Results

FLUX.1-dev (text-to-image):

| Method | Latency(s) | Speedup | PSNR | LPIPS | SSIM | |---|---|---|---|---|---| | Original (50 steps) | 20.9 | 1.0x | — | — | — | | TeaCache (δ=0.3) | 11.4 | 1.83x | 20.76 | 0.211 | 0.810 | | SeaCache (δ=0.3) | 9.4 | 2.22x | 26.29 | 0.106 | 0.893 | | TeaCache (δ=0.6) | 7.1 | 2.94x | 17.21 | 0.348 | 0.714 | | SeaCache (δ=0.6) | 6.4 | 3.27x | 21.33 | 0.226 | 0.798 |

At comparable speedups, SeaCache gains ~5.5 dB PSNR over TeaCache. Qualitatively, it preserves rendered text (e.g., "QUANTUM") and fine scene detail that baselines lose.

HunyuanVideo (text-to-video): 182.6s baseline; SeaCache reaches 90.8s (PSNR 32.39 vs. TeaCache 23.40) and 58.1s (PSNR 26.46 vs. 20.42) — nearly +9 dB at comparable latency. Similar improvements on Wan2.1 1.3B.

Why it works

1. Frequency-domain separation of signal (low-frequency content) from noise (high-frequency residual). 2. Timestep-aware filter aligned with the coarse-to-fine generation rhythm. 3. Zero additional tunable parameters — apply the scheduler-based SEA filtering and run inference.

Limitations and directions

Tested mainly on standard multi-step sampling; benefits shrink on heavily distilled few-step models (FLUX.1-schnell, SDXL-Turbo), though the spectral idea may transfer with filter redesign.
Orthogonal to parallelism (e.g., DistriFusion), efficient attention, and block-level caching; can be combined.
Future work: data-finetuned or content-adaptive filters, learnable end-to-end variants.

Takeaway

SeaCache moves caching decisions from "following the noise" to "following the content" at near-zero cost — a theoretically grounded, training-free 2-3x speedup for image and video diffusion models that is readily deployable in production pipelines.

References

Chung et al. SeaCache: Spectral-Evolution-Aware Cache for Accelerating Diffusion Models. CVPR 2026. arXiv:2602.18993
Ma, Fang & Wang. DeepCache. CVPR 2024.
Chen et al. TeaCache. NeurIPS 2025.
Liu et al. TaylorSeer. NeurIPS 2025.
Kahatapitiya et al. AdaCache. CVPR 2024.