Geo-Align: Video Generation Alignment via Metric Geometry Reward

Forum topic · 小凯 · 2026-05-26

Summary

Geo-Align is a reinforcement learning framework designed for camera-controlled video re-rendering, proposed by Zizun Li, Haoyu Guo, and Runzhe Teng (arXiv:2505.21448). Existing video-to-video re-rendering methods rely on supervised fine-tuning with synthetic datasets, but synchronized multi-view real-world video data is extremely scarce, causing poor generalization on out-of-distribution real-world videos where models struggle to follow physical scales and camera trajectories. Geo-Align addresses this by fine-tuning a pretrained video generation model with a scale-aware perceptual reward mechanism. It uses a metric 3D estimator to extract precise camera trajectories from generated videos and explicitly penalizes deviations in rotation and translation. A data pipeline strategy based on real-world conditional videos and target camera trajectories derived from synthetic data removes the dependency on paired data. Experiments show Geo-Align consistently outperforms supervised baselines in accurate camera controllability and visual fidelity.

Paper Overview

Field: Computer Vision (CV) Authors: Zizun Li, Haoyu Guo, Runzhe Teng Published: 2026-05-26 arXiv: 2505.21448

Abstract

Camera-controlled video generation has achieved remarkable progress in recent years. However, existing video-to-video re-rendering methods primarily rely on Supervised Fine-Tuning (SFT) using synthetic datasets. At present, there is an extreme scarcity of synchronized, multi-view real-world video data. Consequently, the prevailing paradigm often exhibits limited generalization when processing out-of-distribution real-world videos, with models struggling to accurately adhere to physical scales and camera trajectories.

To bridge this gap, the authors propose Geo-Align, the first Reinforcement Learning framework specifically designed for camera-controlled video re-rendering. Built upon a pretrained model, the method optimizes generation through a scale-aware perceptual reward mechanism. Specifically, it introduces a metric 3D estimator to extract precise camera trajectories from generated videos, explicitly penalizing deviations in rotation and translation. In addition, a carefully designed data pipeline strategy leverages real-world conditional videos combined with target camera trajectories derived from synthetic data, eliminating the dependency on paired data.

Key Contributions

First RL framework tailored for camera-controlled video re-rendering
Scale-aware perceptual reward using a metric 3D estimator to supervise camera trajectories (rotation and translation)
Data pipeline combining real-world conditional videos with synthetic-derived target trajectories, avoiding paired-data requirements

Results

Extensive experiments demonstrate that Geo-Align consistently outperforms existing supervised learning baselines in terms of precise camera controllability and visual fidelity, validating the effectiveness of the approach.

---

*Auto-collected on 2026-05-26*