Paper Overview
Field: Computer Vision / Robot Perception
Authors: Tudor Braicu, Joseph DelPreto, Jared Boyer, Mónika Farsang, Maximilian Stölzle, Tim Niklas Seyde, Daniela Rus
Published: 2026-10-27
arXiv: 2610.37619
Abstract
Agile robots require spatiotemporal intelligence to interpret scenes, reason about interactions, and anticipate future events. Yet it remains unclear how well foundation vision models support these capabilities, particularly from a moving egocentric camera, which is a natural view for robots.
To investigate this, the team collected an annotated tennis dataset and trained a suite of vision models on spatiotemporal tasks. The dataset contains egocentric video synchronized with five allocentric videos, estimated global positions, labeled contact times, and annotated stroke types. They then evaluated seven prediction tasks from egocentric images:
- 2D ego player and 3D ball localization
- Future player and ball trajectories
- Forehand or backhand stroke anticipation
- Time to the next ego player ball contact
- 3D ball position at that contact
- Auxiliary depth information substantially improves 3D ball localization, suggesting that inferring metric depth remains a challenge.
- Longer image histories do not improve ball forecasting, indicating that representations exploit scene context rather than rely solely on inter-frame motion.
- A conditional diffusion model generates whole-body kinematic tennis motion from vision encoder predictions on egocentric video at real-time rates, adapting movements as predictions are refined.
Image encoders, video encoders, and vision-language models were compared with jointly trained prediction heads. The best performing models achieve mean current-frame localization errors of 0.14 m for the ego player and 0.54 m for the ball, future forecasting errors of 0.34 m for the ego player and 0.91 m for the ball over 1.1 s, and a mean contact-time prediction error of 0.14 s.