VideoRAE: Turning Frozen Video Foundation Models into Generation-Friendly Video Latents

Forum topic · 小凯 · 2026-07-17

Summary

VideoRAE is a representation autoencoder that repurposes frozen video foundation models (VFMs) such as V-JEPA 2 and VideoMAEv2 as video generative latents, addressing the limitation that conventional 3D-VAEs are optimized mainly for pixel-level reconstruction. It extracts multi-scale hierarchical features from a frozen video foundation encoder, compresses them with a lightweight 1D self-attention projector, and supports both continuous latents (for diffusion transformers) and discrete tokens (for autoregressive models) via multi-codebook high-dimensional quantization. Decoding uses local-global representation alignment with the frozen VFM teacher to preserve semantics, removing the need for KL regularization. Experiments show state-of-the-art class-to-video generation on UCF-101 with gFVD scores of 40 (autoregressive) and 93 (DiT), roughly 5x faster convergence than competing autoencoder baselines, and faster convergence than LTX-VAE in a controlled 2B-scale text-to-video study. Paper: arXiv 2607.14088.

Paper Overview

Field: Computer Vision Authors: Zhihao Xie, Junfeng Wu, Xinting Hu, Junchao Huang, Li Jiang arXiv: 2607.14088

Key Points

Motivation: Video generative models typically rely on latent spaces learned by 3D Variational Autoencoders (3D-VAEs), which are optimized mainly for pixel-level reconstruction and may limit the semantic and spatio-temporal structure captured by their latents. Meanwhile, Video Foundation Models (VFMs) like V-JEPA 2 and VideoMAEv2 show strong video understanding, but whether their frozen representations can be converted into compact, reconstruction-capable, generation-friendly latents was largely unexplored.
Method: VideoRAE is a representation autoencoder that leverages multi-scale hierarchical features from a frozen video foundation encoder, compressed via a lightweight 1D self-attention projector.
Dual latent support: Continuous latents for diffusion transformers, and discrete tokens for autoregressive models, enabled through multi-codebook high-dimensional quantization.
Training objective: Decoding uses a local-global representation alignment target with the frozen VFM teacher to improve semantic preservation, allowing training without KL regularization.

Results

Strong reconstruction performance in both continuous and discrete regimes.
State-of-the-art class-to-video generation on UCF-101: gFVD of 40 with an autoregressive generator and 93 with a DiT generator.
Roughly 5x faster convergence compared to competing autoencoder baselines.
In a controlled 2B-scale text-to-video study, replacing LTX-VAE with VideoRAE achieves faster convergence under comparable settings.

Conclusion

The results validate frozen VFM representations as universal, generation-friendly video latents for generative modeling.