CityRAG: Stepping Into a City via Spatially-Grounded Video Generation

Forum topic · 小凯 · 2026-04-23

Summary

CityRAG is a video generative model that creates 3D-consistent, navigable, spatially grounded environments — effectively simulations of real locations. Presented by researchers including Gene Chou, Noah Snavely, and Philipp Henzler (arXiv 2604.19741), the work addresses a key limitation of existing text-to-video and image-to-video models: while they produce plausible sequences matching prompts, they cannot reliably reconstruct the real world under arbitrary weather conditions and dynamic object configurations, which is essential for autonomous driving and robotics simulation. CityRAG grounds generation to physical scenes by leveraging large corpora of geo-registered data as context, while retaining learned priors for complex motion and appearance changes. A key design choice is the use of temporally unaligned training data, which teaches the model to semantically disentangle the underlying scene from its transient attributes such as weather, lighting, and moving objects. Experiments show that CityRAG generates coherent, physically anchored video sequences lasting minutes, maintains consistent weather and lighting conditions across thousands of frames, achieves loop closure, and navigates complex trajectories to reconstruct real-world geography.

Paper Overview

Field: Computer Vision Authors: Gene Chou, Charles Herrmann, Kyle Genova, Boyang Deng, Songyou Peng, Bharath Hariharan, Jason Y. Zhang, Noah Snavely, Philipp Henzler Published: 2026-04-21 arXiv: 2604.19741

Abstract

We address the problem of generating a 3D-consistent, navigable environment that is spatially grounded: a simulation of a real location. Existing video generative models can produce a plausible sequence that is consistent with a text (T2V) or image (I2V) prompt. However, the capability to reconstruct the real world under arbitrary weather conditions and dynamic object configurations is essential for downstream applications including autonomous driving and robotics simulation.

To this end, we present CityRAG, a video generative model that leverages large corpora of geo-registered data as context to ground generation to the physical scene, while maintaining learned priors for complex motion and appearance changes. CityRAG relies on temporally unaligned training data, which teaches the model to disentangle the underlying scene from its transient attributes in a semantic manner.

Key Results

Experiments demonstrate that CityRAG can generate coherent, minute-long, physically anchored video sequences that:

Maintain weather and lighting conditions over thousands of frames
Achieve loop closure
Navigate complex trajectories to reconstruct real-world geography

---

*Auto-collected on 2026-04-23*