GRACE: Generation-Aware Latent Compression for Efficient Video Diffusion Models

Forum topic · 小凯 · 2026-10-09

Summary

GRACE (Generation-Aware Latent Compression) is a two-stage framework that compresses pretrained video autoencoders while remaining compatible with pretrained Diffusion Transformers (DiTs) for video generation. Highly compressed autoencoders accelerate video diffusion models by giving the DiT fewer tokens, but naive training degrades reconstruction quality, alters the latent distribution, and forces expensive DiT retraining. GRACE freezes the base latents of the pretrained encoder, learns a residual latent that captures information lost under stronger compression, and aligns the compressed latents with the pretrained ones in a frozen DiT feature space, making the autoencoder generation-aware. The DiT is then adapted via lightweight fine-tuning with asymmetric denoising, where the base component is denoised before the residual. Applied to Wan2.1-I2V-14B at 480×832×81, GRACE reduces token count by 8x and latency by 11.1x while matching the uncompressed pipeline's generation quality on VBench. Paper: arXiv 2610.10524.

Paper Overview

Field: Computer Vision (CV)
Authors: Jiyoung Kim, Paul Hyunbin Cho, Jisu Nam, Donghoon Lee, Hyunsung Go, Yeonkyeong Lee, Hansaem Kim, Seungryong Kim
Published: 2026-10-07
arXiv: 2610.10524

Abstract (English)

Highly compressed video autoencoders offer an effective way to accelerate video diffusion models, as the Diffusion Transformer (DiT) operates on far fewer tokens. However, such autoencoders are challenging to train, since a higher compression ratio degrades reconstruction quality and recovering it requires more channels, which is known to slow the convergence of the DiT. The compressed latent also differs from the one the DiT was trained on, so the pretrained DiT must be either retrained from scratch or adapted at considerable cost. Compressing the autoencoder the DiT was trained with appears to preserve compatibility, yet optimizing it for reconstruction alone still shifts the latent away from the distribution the DiT has learned.

To address this, the authors propose GRACE (Generation-Aware Latent Compression for Efficient Video Generation), a two-stage framework that compresses a pretrained video autoencoder while staying compatible with the pretrained DiT.

Key Ideas

Freeze the base latents of the pretrained encoder and learn a residual latent that captures information lost under stronger compression.
Align the compressed latents with the pretrained latents in the frozen DiT's feature space, making the autoencoder generation-aware rather than purely reconstruction-focused.
Adapt the DiT via lightweight fine-tuning and asymmetric denoising, where the base component is denoised before the residual component.

Results

Applied to Wan2.1-I2V-14B at 480×832×81 resolution, GRACE:

Reduces the token count by 8x
Lowers latency by 11.1x
Matches the generation quality of the uncompressed pipeline on VBench