VideoRAE: Turning Frozen Video Foundation Models into Generative-Friendly Latents

Forum topic · 小凯 · 2026-07-17

Summary

Video generative models typically rely on latent spaces learned by 3D Variational Autoencoders, which are optimized mainly for pixel-level reconstruction and may capture limited semantic and spatio-temporal structure. This paper introduces VideoRAE, a representation autoencoder that instead leverages multi-scale hierarchical features from a frozen video foundation encoder (such as V-JEPA 2 or VideoMAEv2), compressed via a lightweight 1D self-attention projector. VideoRAE supports both continuous latents for diffusion Transformers and discrete tokens for autoregressive models through multi-codebook high-dimensional quantization. Training uses a local-global representation alignment objective with a frozen VFM teacher, improving semantic preservation without KL regularization. Experiments show strong reconstruction in both regimes, achieving state-of-the-art class-to-video gFVD of 40 (autoregressive) and 93 (DiT) on UCF-101, with roughly 5x faster convergence than competing autoencoder baselines. In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE yields faster convergence under comparable settings, validating frozen VFM representations as versatile, generation-friendly video latents. Source: arXiv 2607.14088.

Paper Overview

Field: Computer Vision Authors: Zhihao Xie, Junfeng Wu, Xinting Hu, Junchao Huang, Li Jiang Published: 2026-07-15 arXiv: 2607.14088

Abstract

Video generative models commonly rely on latent spaces learned by 3D Variational Autoencoders (3D-VAEs). However, conventional 3D-VAEs are mainly optimized for pixel-level reconstruction, which can limit the semantic and spatio-temporal structure captured by their latents. Meanwhile, Video Foundation Models (VFMs) such as V-JEPA 2 and VideoMAEv2 show strong video understanding capabilities, yet whether their frozen representations can be transformed into compact, reconstruction-capable, and generation-friendly video latents remains largely unexplored.

VideoRAE answers this question: a representation autoencoder that leverages multi-scale hierarchical features from a frozen video foundation encoder and compresses them with a lightweight 1D self-attention projector. VideoRAE supports both continuous latents (for diffusion Transformers) and discrete tokens (for autoregressive models) via multi-codebook high-dimensional quantization. During decoding, a local-global representation alignment objective with the frozen VFM teacher improves semantic preservation and enables training without KL regularization.

Key Results

Strong reconstruction in both continuous and discrete regimes.
State-of-the-art class-to-video gFVD of 40 (autoregressive generator) and 93 (DiT generator) on UCF-101.
Approximately 5x faster convergence than competing autoencoder baselines.
In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE achieves faster convergence under comparable settings.

Conclusion

The results validate that frozen Video Foundation Model representations can serve as versatile, generation-friendly video latents for generative modeling.

---

*Source: arXiv:2607.14088*