Fast Spatial Memory with Elastic Test-Time Training (arXiv 2504.06857)

Summary

Fast Spatial Memory with Elastic Test-Time Training is a computer vision paper (arXiv: 2504.06857) by Ziqiao Ma, Xueyang Yu, and Haoyu Zhen, published on April 9, 2025. The work addresses a key weakness of Large chunk Test-Time Training (LaCT), which performs strongly on long-context 3D reconstruction but uses fully plastic inference-time updates that are prone to catastrophic forgetting and overfitting. The authors propose Elastic Test-Time Training (Elastic TTT), which stabilizes LaCT's fast-weight updates using a Fisher-weighted elastic prior. Building on this architecture, they introduce Fast Spatial Memory (FSM), an efficient and scalable 4D reconstruction model that learns spatiotemporal representations from long observation sequences. Extensive experiments show that FSM enables rapid adaptation over long sequences, achieves high-quality 3D/4D reconstruction with smaller chunk sizes, and mitigates camera interpolation shortcuts.