Latent Spatial Memory for Video World Models: Mirage (arXiv 2506.04879)

Forum topic · 小凯 · 2026-06-10

Summary

This paper introduces latent spatial memory for video world models, a persistent 3D cache that stores scene information directly in the diffusion latent space instead of explicit RGB point clouds. Traditional point cloud memory is computationally expensive, requiring repeated rendering and VAE encoding, and lossy because pixel-space round trips discard rich latent features. The authors propose Mirage, a framework that constructs the memory by lifting latent tokens into 3D through depth-guided back-projection and queries it by synthesizing novel views via direct latent-space warping. This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational overhead of repeated encoding and rendering. Experiments show up to 10.57x end-to-end video generation speedup and 55x memory reduction compared to explicit 3D baselines. Leveraging geometric priors of diffusion models, Mirage achieves state-of-the-art performance on WorldScore and strong reconstruction quality on RealEstate10K. The work was posted on zhichai.net, citing arXiv 2506.04879 by Weijie Wang, Haoyu Zhao, and Yifan Yang, dated June 6, 2025.

Paper Overview

Research Area: Computer Vision Authors: Weijie Wang, Haoyu Zhao, Yifan Yang Published: 2025-06-06 arXiv: 2506.04879

Abstract

Video world models that maintain 3D spatial consistency across generated frames typically rely on explicit point cloud memory constructed in RGB space. This design is both computationally expensive, requiring repeated rendering and VAE encoding, and inherently lossy, as the round trip through pixel space discards rich features of the learned latent representation.

In this paper, the authors introduce latent spatial memory for video world models, a persistent 3D cache that stores scene information directly in the diffusion latent space, avoiding pixel-space reconstruction.

Building on this, they propose Mirage, a latent-space spatial memory framework that:

Constructs memory by lifting latent tokens into 3D via depth-guided back-projection
Queries memory by synthesizing novel views through direct latent-space warping

This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational burden of repeated encoding and rendering.

Results

Up to 10.57x end-to-end video generation speedup compared to explicit 3D baselines
55x reduction in memory footprint
State-of-the-art performance on WorldScore
Strong reconstruction quality on RealEstate10K, leveraging the geometric priors of diffusion models