MotionForesight: Repurposing Video Prediction Models for Future 3D Scene-Flow Prediction

Forum topic · 小凯 · 2026-07-21

Summary

MotionForesight (arXiv:2507.15491) is a research paper by Homanga Bharadhwaj and Yash Jangir in computer vision that repurposes pretrained video prediction models to forecast future 3D motion. Given a short monocular video of human-object interaction, the model predicts future 3D trajectories of points on the manipulated object, without assumptions about object properties. The key insight is that video prediction models already encode rich priors about how objects move during interactions. The authors build a dense 3D tracker on a pretrained video model, generate pseudo-ground-truth tracks from complete clips, then train a lightweight adapter with learned mask latents to convert retrospective tracking into forward prediction, keeping the large video and tracking components frozen. Trained on only 40k human videos with no auxiliary inputs such as language, MotionForesight generalizes to out-of-distribution objects, environments, viewpoints, and interactions, and outperforms substantially larger models trained on over a million videos. The work demonstrates that video priors can be efficiently repurposed into explicit geometric forecasts for embodied intelligence.

Paper Overview

Field: cs.CV Authors: Homanga Bharadhwaj, Yash Jangir Published: 2026-07-21 arXiv: 2507.15491

Abstract

Humans can infer how objects are likely to move from passive observation: a cup may be lifted, a drawer may slide, and a lid may rotate shut. Such predictions expose the physical consequences of interaction needed to act in the real world. MotionForesight studies how to learn this anticipation from ordinary monocular videos of human-object interaction.

Approach

Given a short observed video context, MotionForesight predicts future 3D trajectories for points on the manipulated object. This casts interaction prediction as object-centered 3D motion forecasting without any assumptions on object properties.

The key insight is that video prediction models already encode rich priors about how objects move during human interactions. The method redirects these priors from pixel prediction toward future 3D scene flow:

Start from a dense 3D tracker built on a pretrained video model
Generate pseudo-ground-truth tracks from complete clips
Train the forecaster using only the observed frames
Replace future RGB and geometry with learned mask latents
Train a lightweight adapter to turn the retrospective tracking representation into a forward predictor, while freezing the large video and tracking components

Results

Using just 40k human videos and no auxiliary inputs such as language, MotionForesight generalizes across diverse out-of-distribution objects, environments, viewpoints, and interactions. It also outperforms substantially larger models that use over a million training videos. These results show that video priors can be efficiently re-purposed into explicit geometric forecasts for embodied intelligence.

---

*Source: arXiv:2507.15491, auto-collected 2026-07-21*