Online Neural Space Time Memory for Dynamic Novel View Synthesis

Forum topic · 小凯 · 2026-07-18

Summary

This paper introduces an online neural space-time memory framework for novel view synthesis from multi-view streaming video. The core challenge is balancing persistent long-horizon memory—needed to reconstruct temporarily occluded regions—against strict real-time constraints. Test-Time Training (TTT) provides a strong memory mechanism, but standard models require gradient-based memory updates at every frame to adapt to changing motion in dynamic scenes, making real-time use infeasible and risking instability over long contexts. Since memory updates are more expensive than memory application and video content is largely redundant, the authors decouple the frequencies of the two processes: memory updates occur periodically, while memory is applied on every frame using cross-view attention to handle warping between past memory states and the current frame. Two mechanisms lock in historical context: an auxiliary memory loss that forces persistent internalization of the scene, and a memory caching strategy that regularizes active weights to prevent catastrophic drift. The method achieves real-time, state-of-the-art performance on dynamic human motion scenes and minute-scale online memorization. Authors include Baback Elmieh, Stephen Lombardi, Steven M. Seitz, and Xuan Luo; paper: arXiv 2607.15271.

Online Neural Space Time Memory for Dynamic Novel View Synthesis

Paper Overview

Field: Computer Vision (CV)
Authors: Baback Elmieh, Lynn Tsai, Zeman Li, Srinivas Kaza, Tiancheng Sun, Gabor Csapo, Ali Behrouz, Yuan Deng, Stephen Lombardi, Steven M. Seitz, Xuan Luo
arXiv: 2607.15271

Summary

Online novel view synthesis from multi-view streaming videos faces a fundamental trade-off: maintaining a persistent, long-horizon memory to reconstruct temporarily occluded regions while operating under strict real-time constraints. While Test-Time Training (TTT) offers a powerful memory mechanism, standard models mandate gradient-based memory updates at every frame to adapt to the changing motion in dynamic scenes. The computational cost of heavy memory updates precludes real-time application and can lead to instability over long contexts.

Given that memory updates are more demanding than memory application, and video content is largely redundant, the authors propose to decouple the frequencies of these two processes:

Periodic memory updates instead of per-frame gradient updates.
Per-frame memory application using cross-view attention to manage the warping between previous memory states and the current frame.

To lock in historical context, two key mechanisms are introduced:

An auxiliary memory loss that enforces persistent internalization of the scene.
A memory caching strategy that regularizes active weights to prevent catastrophic drift.

Results

The method demonstrates real-time, state-of-the-art performance on dynamic human motion scenes as well as minute-scale online memorization.

---

*Auto-collected on 2026-07-18.*

Online Neural Space Time Memory for Dynamic Novel View Synthesis

Summary

Online Neural Space Time Memory for Dynamic Novel View Synthesis

Paper Overview

Summary

Results

Tags