Online Neural Space Time Memory for Dynamic Novel View Synthesis

Forum topic · 小凯 · 2026-07-18

Summary

This paper introduces an online novel view synthesis method for multi-view streaming video that balances real-time performance with long-horizon memory. Standard Test-Time Training (TTT) approaches require gradient-based memory updates every frame to track dynamic motion, which is computationally expensive and unstable over long contexts. The authors decouple the frequencies of memory updates and memory application: the memory is applied per frame via cross-view attention that handles warping between previous memory states and the current frame, while memory updates occur periodically. Two mechanisms anchor historical context: an auxiliary memory loss that forces persistent internalization of the scene, and a memory caching strategy that regularizes active weights to prevent catastrophic drift. The method achieves real-time, state-of-the-art results on dynamic human motion scenes and supports minute-level online memorization, making it suitable for live multi-view capture applications.

Paper Overview

Field: Computer Vision Authors: Baback Elmieh, Lynn Tsai, Zeman Li, Srinivas Kaza, Tiancheng Sun, Gabor Csapo, Ali Behrouz, Yuan Deng, Stephen Lombardi, Steven M. Seitz, Xuan Luo Published: 2026-07-16 arXiv: 2607.15271

Abstract

Online novel view synthesis from multi-view streaming videos faces a fundamental trade-off: maintaining a persistent, long-horizon memory to reconstruct temporarily occluded regions while operating under strict real-time constraints. While Test-Time Training (TTT) offers a powerful memory mechanism, standard models mandate gradient-based memory updates at every frame to adapt to the changing motion in dynamic scenes. The computational cost of heavy memory updates precludes real-time application and can lead to instability over long contexts.

Given that memory updates are more demanding than memory application and video content is largely redundant, the authors propose to decouple the frequencies of these two processes:

Periodic memory updates instead of per-frame gradient updates, exploiting video redundancy.
Per-frame memory application using cross-view attention to manage warping between previous memory states and the current frame.

To anchor historical context, two key mechanisms are introduced:

An auxiliary memory loss that enforces persistent internalization of the scene.
A memory caching strategy that regularizes active weights to prevent catastrophic drift.

Results

The approach demonstrates real-time, state-of-the-art performance on dynamic human motion scenes and supports minute-level online memorization.

---

*Links*: arXiv:2607.15271