[论文] Repurposing Video Diffusion Transformers for Cross-View Temporal Objec...

论文概要

研究领域: CV / 视频理解
作者: Siyoon Jin, Dahyun Chung, Honggyu An, Sangbeom Lim, Wonjin Nam, Jaewoo Jung, WonJun Moon, Seungryong Kim
发布时间: 2026-10-27
arXiv: 2610.37633

中文摘要

跨视角物体对应需要在不同视角中识别同一物体。现有方法多依赖逐帧匹配或顺序跟踪，在外观变化或遮挡下容易传播对应误差。本文引入跨视角时间物体对应任务，要求从query mask出发同时保持跨视角和跨时间的物体身份。提出XTrack——一个统一框架，将预训练视频DiT的语义和时空先验重新用于RGB到mask的潜在传输，联合建模跨视角匹配和时间跟踪。但原始视频DiT不保证准确的跨视角定位，常关注语义相似但错误的物体。为此引入焦点跨视角注意力对齐、困难负例条件和局部上下文裁剪策略。此外，还构建了伪标签策划管线，将稀疏Ego-Exo4D标注转为密集监督以支持视频级训练。

原文摘要

Cross-view object correspondence identifies the same object across views. However, existing methods often rely on frame-wise matching or sequential tracking, which can propagate correspondence errors under appearance changes or occlusions. Therefore, we introduce cross-view temporal object correspondence, which requires preserving object identity across both views and time from a query mask. To address this, we propose XTrack, a unified framework which repurposes a pretrained video DiT with semantic and spatiotemporal priors for RGB-to-mask latent transport, jointly modeling cross-view matching and temporal tracking. However, naive video DiTs do not guarantee accurate cross-view grounding and often attend to semantically similar yet incorrect objects across views. Thus, we introduce focal cross-view attention alignment, hard negative conditioning, and local context crop strategies to ensure reliable grounding. Finally, beyond these architectural enhancements, we also tackle the lack of dense annotation by introducing a pseudo-label curation pipeline that converts sparse Ego-Exo4D annotations into dense supervision to enable effective video-level training. Our results demonstrate the effectiveness of XTrack as a unified framework, ensuring robust temporal consistency and identity preservation.

自动采集于 2026-10-29

#论文 #arXiv #AI #小凯