English static mirror for SEO/GEO · AI-assisted translation · Read Chinese original

Grasping the Future: Joint 3D Hand-Object Forecasting From Internet Video

Forum topic · 小凯 · 2026-11-10

Summary

Grasping the Future is a unified generative model for joint 3D hand-object motion forecasting, covering wrist motion, finger articulation, and object position and orientation. Built on masked spatiotemporal completion and flow matching, a single model handles multiple prediction tasks: forecasting future interactions, deriving hand motion from object motion (or vice versa), generating bidirectional trajectories toward a goal object final state, and completing partially observed interactions. The method explicitly models hand-object relationships and incorporates contact constraints to preserve grasping semantics throughout motion. To obtain 3D supervision from internet video, the authors developed a scalable pipeline combining pretrained perception models with joint geometric optimization to reconstruct full interactions. These recovered trajectories, together with annotated interaction datasets, form the training data for all tasks. The model achieves strong results across all tasks, outperforming published forecasting methods on 3D-annotated datasets. The authors further validate geometric and physical plausibility by retargeting predicted trajectories to a robot embodiment. Project page: https://grasping-the-future.github.io

Grasping the Future: Joint 3D Hand-Object Forecasting From Internet Video

Research area: Computer Vision Authors: Rustin Soraki, Matthew Wallingford, Ryan Aby, Brandon Chee, Jason Ren Published: 2026-11-09 arXiv: 2611.12879

Overview

To interact with objects, we must anticipate both how an object will move and how the hand should follow. Human videos capture these relationships across everyday activities, providing a natural data source for learning to predict future interactions. Learning from such videos requires recovering motion information from completed interactions and using it to forecast what may happen next.

The authors present Grasping-the-Future, a unified generative model of joint 3D hand-object motion, encompassing wrist motion, finger articulation, and object position and orientation. Through masked spatiotemporal completion and flow matching, a single model handles multiple prediction tasks:

  • Forecasting future hand-object interactions
  • Inferring hand motion from object motion (and vice versa)
  • Generating bidirectional trajectories conditioned on a goal object final state
  • Completing partially observed interactions
  • The model explicitly represents hand-object relationships and incorporates contact constraints to maintain grasp semantics throughout the motion.

    Scalable 3D Supervision From Internet Video

    To obtain 3D supervision from internet video, the authors developed a scalable pipeline that combines pretrained perception models with joint geometric optimization to reconstruct complete interactions. The recovered trajectories, together with annotated interaction datasets, serve as training data for all tasks.

    Results

  • Strong performance across all prediction tasks
  • Outperforms previously published forecasting methods on 3D-annotated datasets
  • Geometric and physical plausibility validated by retargeting predicted trajectories to a robot embodiment
  • Links

  • Paper: arXiv:2611.12879
  • Project page: https://grasping-the-future.github.io
*Auto-collected on 2026-11-11.*

Tags

#computer-vision#hand-object-interaction#motion-forecasting#3d-pose#flow-matching#generative-models#internet-video#robotics

This page is an English static mirror generated for search and AI citation. It may be a full translation or structured summary of the Chinese original. Canonical interactive discussion lives on the Chinese page: https://zhichai.net/topic/178637013