[论文] V-RAE: Rethinking Video Latent Spaces for Generation
论文概要
研究领域: CV 作者: Minghui Guo, Shengqiong Wu, Hao Fei 发布时间: 2026-08-13 arXiv: 2608.13556中文摘要
潜变量视频生成依赖自编码器定义一个紧凑空间供生成模型操作。尽管视频自编码器架构已大幅演进，其潜变量空间仍主要针对像素级重建优化，提供的高级语义组织有限。然而，重建最优的潜变量空间未必适合生成建模。我们提出V-RAE，一种视频表示自编码器，在冻结的视觉基础模型表示之上构建紧凑的生成潜变量。轻量级时间池化模块去除时间冗余同时保留语义结构，视频解码器从压缩特征重建连续运动。我们使用四种代表性冻结编码器在视频重建、语义探测和类别条件生成上评估V-RAE。V-RAE在K600上达到2.13 rFVD，超越所有评估的大规模预训练视频VAE。其潜变量比传统视频分词器潜变量保留更多语义信息。在匹配的生成设置下，我们的最佳变体在UCF101和K600上分别达到117.86和19.16的gFVD分数，同时收敛速度提升高达6倍。我们进一步证明，仅靠重建质量不足以表征生成效用，并引入tFVD，一种与下游生成质量更可靠相关的时间一致性诊断指标。除视频生成外，V-RAE在Cityscapes上的未来视频预测也优于Wan 2.2 VAE潜变量空间。综合实验表明，冻结的语义表示可以支持视频重建、生成和预测建模。项目页面：https://v-rae.github.io/原文摘要
Latent video generation relies on autoencoders to define a compact space in which generative models operate. Although video自编码器architectures have evolved substantially, their latent spaces are still optimized primarily for pixel-level reconstruction and provide limited high-level semantic organization. A reconstruction-optimal latent space, however, need not be well suited to generative modeling. We propose V-RAE, a video representation autoencoder that builds compact generative latents on top of frozen vision foundation model representations. A lightweight temporal pooling module removes temporal redundancy while preserving semantic structure, and a video decoder reconstructs continuous motion from the compressed features. We evaluate V-RAE with four representative frozen encoders on video reconstruction, semantic probing, and class-conditional generation. V-RAE achieves 2.13 rFVD on K600, outperforming all evaluated large-scale pretrained video VAEs. Its latents retain substantially more semantic information than conventional video tokenizer latents. Under matched generation settings, our best variant achieves gFVD scores of 117.86 and 19.16 on UCF101 and K600, respectively, while converging up to 6x faster}. We further show that reconstruction quality alone is insufficient to characterize generative utility and introduce tFVD, a temporal-coherence diagnostic that correlates more reliably with downstream generation quality. Beyond video generation, V-RAE also improves future video prediction on Cityscapes over the Wan 2.2 VAE latent space under matched prediction settings. Taken together, the experiments show that frozen semantic representations can support video reconstruction, generation, and predictive modeling. The project page: https://v-rae.github.io/.--- *自动采集于 2026-08-15*
#论文 #arXiv #CV #小凯