Overview
Field: Computer Vision Authors: Assaf Singer, Maroun Khriesh, Noam Rotstein, Yoni Kasten, Or Litany Published: 2026-10-30 arXiv: 2610.42136
Full Translation
Dense 3D tracking from monocular video requires establishing point correspondences over time, estimating their depth, and inferring their trajectories during occlusion. Existing methods typically learn these tasks jointly, a demanding objective given the scarcity of 4D supervision. The authors show that strong pretrained 2D trackers and 3D geometry models provide an effective foundation: lifted trajectories, with occluded segments simply interpolated, already rival dedicated 3D trackers using the same geometry.
However, lifted trajectories inherit correspondence and depth errors, and under occlusion, their depth comes from the occluder rather than the tracked point. ReLift is introduced as an efficient feed-forward framework that recasts dense 3D tracking as geometric trajectory repair and completion. At its core is a trajectory corrector that alternates attention along trajectories and across trajectories, preserving reliable observations, repairing corrupted locations, and completing missing motion.
Using only geometric inputs enables training directly on 3D trajectories with simulated lifted errors, decoupling training from images and from any specific tracker or geometry model. Concentrating 4D supervision on trajectory refinement yields a reusable model that improves lifted trajectories from diverse trackers and geometry providers without retraining, so it directly benefits from future advances in both components.
On WorldTrack, ReLift achieves state-of-the-art average precision, outperforming the strongest baseline by 2.1 APD on all points and 3.0 on dynamic points; a faster configuration remains competitive with leading 3D trackers while running faster than 4RC. More examples and code are available on the project's supplementary page.
Links
- arXiv: https://arxiv.org/abs/2610.42136