[论文] Perceiving and Predicting Dynamic Sports: Spatiotemporal Intelligence ...

论文概要

研究领域: CV / 机器人感知
作者: Tudor Braicu, Joseph DelPreto, Jared Boyer, Mónika Farsang, Maximilian Stölzle, Tim Niklas Seyde, Daniela Rus
发布时间: 2026-10-27
arXiv: 2610.37619

中文摘要

敏捷机器人需要时空智能来解释场景、推理交互并预测未来事件。然而，基础视觉模型在多大程度上支持这些能力仍不清楚，特别是从移动的第一人称视角——这是机器人的自然视角。为研究此问题，团队收集了带标注的网球数据集，包含与五个外视角视频同步的第一人称视频、估计的全局位置、标注的接触时间和击球类型。然后评估了七个从第一人称图像出发的预测任务：2D自我球员和3D球定位、未来球员和球轨迹、正手或反手预判、到下一次球接触的时间、以及接触时的3D球位置。最佳模型的当前帧定位误差：自我球员0.14m、球0.54m；未来1.1秒预测误差：球员0.34m、球0.91m；接触时间预测误差0.14s。辅助深度信息显著改善3D球定位，表明度量深度推断仍具挑战。更长的图像历史不能改善球的预测，说明表示利用的是场景上下文而非帧间运动。

原文摘要

Agile robots require spatiotemporal intelligence to interpret scenes, reason about interactions, and anticipate future events. Yet it remains unclear how well foundation vision models support these capabilities, particularly from a moving egocentric camera, which is a natural view for robots. To investigate this, we collect an annotated tennis dataset and train a suite of vision models on spatiotemporal tasks. Our dataset contains egocentric video synchronized with five allocentric videos, estimated global positions, labeled contact times, and annotated stroke types. We then evaluate seven prediction tasks from egocentric images: 2D ego player and 3D ball localization, future player and ball trajectories, forehand or backhand stroke anticipation, time to the next ego player ball contact, and 3D ball position at that contact. We compare image encoders, video encoders, and vision-language models with jointly trained prediction heads. The best performing models achieve mean current-frame localization errors of 0.14 m for the ego player and 0.54 m for the ball, future forecasting errors of 0.34 m for the ego player and 0.91 m for the ball over 1.1 s, and a mean contact-time prediction error of 0.14 s. Auxiliary depth information substantially improves 3D ball localization, suggesting that inferring metric depth remains a challenge. Longer image histories do not improve ball forecasting, indicating that representations exploit scene context rather than rely solely on inter-frame motion. Finally, we propose a conditional diffusion model to generate whole-body kinematic tennis motion from vision encoder predictions on egocentric video at real-time rates, adapting movements as predictions are refined. These results move towards a deeper understanding of how foundation vision models can support real-time dynamic robots using egocentric video.

自动采集于 2026-10-29

#论文 #arXiv #AI #小凯