Summary
This paper (arXiv:2607.15271) addresses real-time novel view synthesis from multi-view streaming video, where maintaining a persistent long-horizon memory for reconstructing temporarily occluded regions conflicts with strict real-time constraints. The authors observe that Test-Time Training (TTT) offers a strong memory mechanism, but standard gradient-based memory updates every frame are computationally prohibitive and unstable over long contexts. Since memory updates are more expensive than memory application and video content is largely redundant, they decouple the two processes' frequencies: the model performs periodic memory updates while applying existing memory per frame, using cross-view attention to manage warping between prior memory states and the current frame. Two mechanisms lock historical context: an auxiliary memory loss enforcing persistent internalization of the scene, and a memory cache strategy that regularizes active weights against catastrophic drift. The method achieves real-time, state-of-the-art performance on dynamic human motion scenes and minute-scale online memorization. Authors include researchers from a computer vision team (Baback Elmieh, Stephen Lombardi, Steven M. Seitz, Xuan Luo, and others), published July 2026.
Paper Overview
- Field: Computer Vision
- Authors: Baback Elmieh, Lynn Tsai, Zeman Li, Srinivas Kaza, Tiancheng Sun, Gabor Csapo, Ali Behrouz, Yuan Deng, Stephen Lombardi, Steven M. Seitz, Xuan Luo
- Published: 2026-07-16
- arXiv: 2607.15271
Summary
Online novel view synthesis from multi-view streaming videos faces a fundamental trade-off: maintaining a persistent, long-horizon memory to reconstruct temporarily occluded regions while operating under strict real-time constraints.
While Test-Time Training (TTT) offers a powerful memory mechanism, standard models mandate gradient-based memory updates at every frame to adapt to changing motion in dynamic scenes. The computational cost of heavy memory updates precludes real-time application and can lead to instability over long contexts.
Given that memory updates are more demanding than memory application, and that video content is largely redundant, the authors propose decoupling the frequencies of the two processes:
- Perform periodic memory updates, while applying memory on a per-frame basis.
- Use cross-view attention to manage warping between prior memory states and the current frame.
Two key mechanisms lock in historical context:
1. An auxiliary memory loss that enforces persistent internalization of the scene.
2. A memory cache strategy that regularizes active weights to prevent catastrophic drift.
Results
The approach demonstrates real-time, state-of-the-art performance on dynamic human motion scenes as well as minute-scale online memorization.
---
*Auto-collected on 2026-07-18*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178433590