XTrack: Repurposing Video Diffusion Transformers for Cross-View Temporal Object Correspondence

Forum topic · 小凯 · 2026-10-28

Summary

XTrack is a unified framework for cross-view temporal object correspondence, a new task that requires identifying and preserving the same object's identity across both camera views and time, starting from a query mask. Existing methods rely on frame-wise matching or sequential tracking, which propagates errors under appearance changes or occlusions. XTrack repurposes a pretrained video Diffusion Transformer (DiT) with semantic and spatiotemporal priors for RGB-to-mask latent transport, jointly modeling cross-view matching and temporal tracking. To address unreliable cross-view grounding, the authors introduce focal cross-view attention alignment, hard negative conditioning, and local context crop strategies. A pseudo-label curation pipeline converts sparse Ego-Exo4D annotations into dense supervision for video-level training. Results show robust temporal consistency and identity preservation.

Paper Overview

Research Area: Computer Vision / Video Understanding

Authors: Siyoon Jin, Dahyun Chung, Honggyu An, Sangbeom Lim, Wonjin Nam, Jaewoo Jung, WonJun Moon, Seungryong Kim

arXiv: 2610.37633

Abstract

Cross-view object correspondence identifies the same object across views. However, existing methods often rely on frame-wise matching or sequential tracking, which can propagate correspondence errors under appearance changes or occlusions. Therefore, we introduce cross-view temporal object correspondence, which requires preserving object identity across both views and time from a query mask. To address this, we propose XTrack, a unified framework which repurposes a pretrained video DiT with semantic and spatiotemporal priors for RGB-to-mask latent transport, jointly modeling cross-view matching and temporal tracking. However, naive video DiTs do not guarantee accurate cross-view grounding and often attend to semantically similar yet incorrect objects across views. Thus, we introduce focal cross-view attention alignment, hard negative conditioning, and local context crop strategies to ensure reliable grounding. Finally, beyond these architectural enhancements, we also tackle the lack of dense annotation by introducing a pseudo-label curation pipeline that converts sparse Ego-Exo4D annotations into dense supervision to enable effective video-level training. Our results demonstrate the effectiveness of XTrack as a unified framework, ensuring robust temporal consistency and identity preservation.

Key Contributions

New task: Cross-view temporal object correspondence, jointly preserving object identity across views and time from a query mask
Unified architecture: XTrack repurposes a pretrained video Diffusion Transformer for RGB-to-mask latent transport, combining cross-view matching and temporal tracking
Grounding strategies: Focal cross-view attention alignment, hard negative conditioning, and local context crop to prevent attending to semantically similar but incorrect objects
Data: Pseudo-label curation pipeline converting sparse Ego-Exo4D annotations into dense supervision for video-level training

*Auto-collected on 2026-10-29.*