Latent Spatial Memory for Video World Models: Mirage Achieves 10.57x Speedup

Forum topic · 小凯 · 2026-06-10

Summary

A paper (arXiv 2506.04879) by Weijie Wang, Haoyu Zhao, and Yifan Yang, published June 6, 2025, introduces latent spatial memory for video world models. Conventional approaches maintain 3D spatial consistency across generated frames using explicit point cloud memory built in RGB space, which is computationally expensive due to repeated rendering and VAE encoding, and inherently lossy because pixel-space round trips discard rich latent features. The proposed method stores scene information directly in the diffusion latent space as a persistent 3D cache, avoiding pixel-space reconstruction. The authors present Mirage, a framework that builds memory by lifting latent tokens into 3D via depth-guided back-projection and queries it by synthesizing novel views through direct latent-space warping. Experiments show up to 10.57x end-to-end video generation speedup and 55x memory reduction versus explicit 3D baselines, with state-of-the-art WorldScore and strong reconstruction quality on RealEstate10K.

Overview

Research area: Computer Vision Authors: Weijie Wang, Haoyu Zhao, Yifan Yang Published: 2025-06-06 arXiv: 2506.04879

Abstract

Video world models that maintain 3D spatial consistency across generated frames typically rely on explicit point cloud memory constructed in RGB space. This design is both computationally expensive, requiring repeated rendering and VAE encoding, and inherently lossy, as the round trip through pixel space discards rich features of the learned latent representation.

In this paper, the authors introduce latent spatial memory for video world models: a persistent 3D cache that stores scene information directly in the diffusion latent space, avoiding pixel-space reconstruction.

Mirage Framework

Building on this idea, the authors propose Mirage, a latent-space spatial memory framework that:

Constructs the memory by lifting latent tokens into 3D via depth-guided back-projection
Queries the memory by synthesizing novel views through direct latent-space warping

This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational burden of repeated rendering and encoding.

Results

Up to 10.57x end-to-end video generation speedup over explicit 3D baselines
55x reduction in memory footprint
State-of-the-art performance on WorldScore
Strong reconstruction quality on RealEstate10K