This forum post introduces the paper V-RAE: Rethinking Video Latent Spaces for Generation (CV), by Minghui Guo, Shengqiong Wu, and Hao Fei, posted to arXiv as 2608.13556.
Key Points
- Problem: Latent video generation relies on autoencoders to define a compact space for generative models. Existing video autoencoder latent spaces are optimized mainly for pixel-level reconstruction, offering limited high-level semantic organization — and a reconstruction-optimal latent space is not necessarily well suited to generative modeling.
- Approach: V-RAE (video representation autoencoder) builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features.
- Evaluation: Tested with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation.
- Achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs.
- Latents retain substantially more semantic information than conventional video tokenizer latents.
- Under matched generation settings, the best variant achieves gFVD of 117.86 on UCF101 and 19.16 on K600, converging up to 6x faster.
- Reconstruction quality alone is shown to be insufficient for characterizing generative utility; the authors introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality.
- Beyond generation, V-RAE improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched settings.
- Project page: https://v-rae.github.io/
- Paper: https://arxiv.org/abs/2608.13556
Results
Conclusion
The experiments demonstrate that frozen semantic representations from vision foundation models can support video reconstruction, generation, and predictive modeling.