[论文] Back to Pixels: Lifting Latent Video Diffusion into Pixel Space

论文概要

研究领域: CV
作者: Yanke Song, Shuo Chen, Chris Rockwell, Guande He, Nan Liu, Desai Xie, Tianpei Gu, Yong-Hyun Park, Aryan V S, Dennis Yuren Menn, Ming-Hsuan Yang, Xun Huang
发布时间: 2026-11-12
arXiv: 2611.17336

中文摘要

前沿视频生成模型在 VAE 潜在空间中工作，该空间同时在空间和时间维度压缩视频。这种压缩让训练变得可行，但如今限制了高分辨率、可控和交互式生成——分辨率翻倍会让 token 数量变为四倍，条件只能作用于帧组而非单帧，每次响应都必须等待整个多帧块完成。我们提出 PixVid，一种简单而有效的方法，将预训练的潜在视频模型提升到直接在像素空间逐帧生成。PixVid 保留扩散 Transformer，用卷积头替换 VAE 解码器，将输出渲染为像素。我们发现，预测干净视频而非速度（velocity），配合更大的噪声偏移，是质量的关键。PixVid 从 Wan2.1-14B 微调而来，仅用约 130 万条训练剪辑便达到了与潜在基线相当的水平。在像素空间逐帧工作，它以相同的每帧 token 数实现了 2 倍更高分辨率的生成，在快速运动下保持深度条件视频编辑对齐，并在下一帧即时响应每个动作。

原文摘要

Frontier video generation models work in a VAE latent space that compresses video in both space and time. This compression made training tractable, but now limits high-resolution, controllable, and interactive generation -- doubling resolution quadruples token count, conditions apply to groups of frames rather than individual ones, and each response must wait for the rest of its multi-frame block. We present PixVid, a simple yet effective method that lifts a pretrained latent video model to directly generate every frame in pixel space. PixVid keeps the diffusion transformer and replaces the VAE decoder with a convolutional head that renders its outputs into pixels. We find that predicting clean video rather than velocity, together with a larger noise shift, is key to quality. Finetuned fro...

自动采集于 2026-11-14

#论文 #arXiv #CV #小凯