Grasping the Future: Joint 3D Hand-Object Forecasting From Internet Video
Research area: Computer Vision Authors: Rustin Soraki, Matthew Wallingford, Ryan Aby, Brandon Chee, Jason Ren Published: 2026-11-09 arXiv: 2611.12879
Overview
To interact with objects, we must anticipate both how an object will move and how the hand should follow. Human videos capture these relationships across everyday activities, providing a natural data source for learning to predict future interactions. Learning from such videos requires recovering motion information from completed interactions and using it to forecast what may happen next.
The authors present Grasping-the-Future, a unified generative model of joint 3D hand-object motion, encompassing wrist motion, finger articulation, and object position and orientation. Through masked spatiotemporal completion and flow matching, a single model handles multiple prediction tasks:
- Forecasting future hand-object interactions
- Inferring hand motion from object motion (and vice versa)
- Generating bidirectional trajectories conditioned on a goal object final state
- Completing partially observed interactions
- Strong performance across all prediction tasks
- Outperforms previously published forecasting methods on 3D-annotated datasets
- Geometric and physical plausibility validated by retargeting predicted trajectories to a robot embodiment
- Paper: arXiv:2611.12879
- Project page: https://grasping-the-future.github.io
The model explicitly represents hand-object relationships and incorporates contact constraints to maintain grasp semantics throughout the motion.
Scalable 3D Supervision From Internet Video
To obtain 3D supervision from internet video, the authors developed a scalable pipeline that combines pretrained perception models with joint geometric optimization to reconstruct complete interactions. The recovered trajectories, together with annotated interaction datasets, serve as training data for all tasks.