V-RAE: Rethinking Video Latent Spaces for Generation

Forum topic · 小凯 · 2026-08-15

Summary

V-RAE is a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations instead of optimizing purely for pixel-level reconstruction. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. Evaluated with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation, V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs, with latents retaining more semantic information than conventional video tokenizer latents. In matched generation settings, its best variant reaches gFVD scores of 117.86 on UCF101 and 19.16 on K600 while converging up to 6x faster. The authors show reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space. Paper: arXiv 2608.13556; project page: https://v-rae.github.io/

Overview

Field: Computer Vision
Authors: Minghui Guo, Shengqiong Wu, Hao Fei
Published: 2026-08-13
arXiv: 2608.13556
Project page: https://v-rae.github.io/

Abstract

Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video autoencoder architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. The authors propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features.

Key Findings

Reconstruction: V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs (tested with four representative frozen encoders).
Semantics: Its latents retain substantially more semantic information than conventional video tokenizer latents.
Generation: Under matched generation settings, the best variant achieves gFVD scores of 117.86 on UCF101 and 19.16 on K600, while converging up to 6x faster.
New metric: Reconstruction quality alone is insufficient to characterize generative utility; the paper introduces tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality.
Prediction: V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched settings.

Conclusion

The experiments demonstrate that frozen semantic representations from vision foundation models can support video reconstruction, generation, and predictive modeling.

---

*Auto-collected on 2026-08-15.*