Online Neural Space Time Memory for Dynamic Novel View Synthesis
Paper: arXiv:2607.15271 Authors: Baback Elmieh, Lynn Tsai, Zeman Li, Srinivas Kaza, Tiancheng Sun, Gabor Csapo, Ali Behrouz, Yuan Deng, Stephen Lombardi, Steven M. Seitz, Xuan Luo Posted: 2026-07-16 (auto-collected 2026-07-18)
Abstract (Original)
Online novel view synthesis from multi-view streaming videos faces a fundamental trade-off: maintaining a persistent, long-horizon memory to reconstruct temporarily occluded regions while operating under strict real-time constraints. While Test-Time Training (TTT) offers a powerful memory mechanism, standard models mandate gradient-based memory updates at every frame to adapt to the changing motion in dynamic scenes. The computational cost of heavy memory updates precludes real-time application and can lead to instability over long contexts.
Key Idea
Given that memory updates are more demanding than memory application, and that video content is largely redundant, the authors propose to decouple the frequencies of the two processes:
- Perform periodic memory updates instead of per-frame gradient updates.
- Apply memory on every frame using cross-view attention to manage warping between previous memory states and the current frame.
- Auxiliary memory loss: forces persistent internalization of the scene to lock in historical context.
- Memory caching strategy: regularizes active weights to prevent catastrophic drift over long sequences.
- arXiv: https://arxiv.org/abs/2607.15271
Key Mechanisms
Results
The method demonstrates real-time, state-of-the-art performance on dynamic human motion scenes and supports minute-scale online memorization.