Paper Overview
Field: Computer Vision Authors: Shengxiang Ji, Boyang Wang, Haiyang Xu, Bingnan Li, Yucheng Mao, Zeyuan Chen, Xiaojun Shan, Xiang Zhang, Gang Hua, Jianwen Xie, Zezhou Cheng, Zhuowen Tu arXiv: 2609.38146
Abstract
We introduce LIFT, a unified image-to-video generation framework that complements camera control with Layout-In-FuTure control, enabling users to specify what should appear in a future view and where it should appear. This addresses a practical need in controllable video generation: given an initial image, users often care not only about how the camera moves, but also about what the scene should look like at key future moments, especially the final frame. Existing camera controls specify viewpoint trajectories, while text prompts provide only coarse semantic guidance; neither precisely determines the content and spatial layout of future views. This limitation becomes particularly pronounced under large viewpoint changes, where the camera reveals regions that are not visible in the first frame. LIFT therefore uses final-frame layout as an explicit control signal for the desired future scene. Since learning from such sparse layout guidance is far more challenging than learning with dense per-frame layout conditioning, we introduce Online Policy Self-Distillation (OPSD) to transfer the control capability of a dense-layout teacher to a final-frame-layout student. We also construct the LIFT-Vista dataset, containing camera- and time-consistent layout annotations under large viewpoint changes. Experiments show that LIFT outperforms other methods in video quality, future-layout controllability, and camera controllability.
--- *Auto-collected on 2026-10-01.*