ThinkJEPA: VLM-Guided Latent World Models for Long-Horizon Video Prediction

Forum topic · 小凯 · 2026-03-25

Summary

ThinkJEPA is a framework that empowers latent world models with large vision-language model (VLM) reasoning. Latent world models like V-JEPA2 forecast future world states from video, but dense prediction over short observation windows limits temporal context and biases models toward local, low-level extrapolation. VLMs offer strong semantic grounding by reasoning over uniformly sampled frames, yet suffer from sparse sampling, a language-output bottleneck that compresses fine-grained interaction states into text, and data-regime mismatch on small action-conditioned datasets. ThinkJEPA combines both via dual temporal pathways: a dense JEPA branch for fine-grained motion and interaction cues, and a uniformly sampled VLM thinking branch providing long-horizon semantic guidance. A hierarchical pyramid representation extraction module aggregates multi-layer VLM features into guidance signals compatible with latent prediction. Experiments on hand manipulation trajectory prediction show it outperforms strong VLM-only and JEPA-predictor baselines.

Overview

Field: NLP / World Models
Authors: Haichao Zhang, Yijiang Li, Shwai He, Tushar Nagarajan, Mingfei Chen, Jianglin Lu, Ang Li, Yun Fu
Published: 2026-03-23
arXiv: 2603.22281

Key Points

Recent latent world models (e.g., V-JEPA2) show promise in forecasting future world states from video observations.
However, dense prediction from short observation windows limits temporal context and biases predictors toward local, low-level extrapolation, making long-horizon semantics hard to capture.
Vision-language models (VLMs) provide strong semantic grounding via reasoning over uniformly sampled frames, but are not ideal standalone dense predictors due to:
compute-driven sparse sampling,
a language-output bottleneck that compresses fine-grained interaction states into text-oriented representations,
data-regime mismatch when adapting to small action-conditioned datasets.

Method

ThinkJEPA is a VLM-guided, JEPA-style latent world modeling framework combining dense frame dynamics with long-horizon semantic guidance through dual temporal pathways:

A dense JEPA branch for fine-grained motion and interaction cues.
A VLM thinking branch with a larger temporal span for rich knowledge guidance via uniformly sampled frames.

To transfer the VLM's progressive reasoning signals, the authors introduce a hierarchical pyramid representation extraction module that aggregates multi-layer VLM representations into guidance features compatible with latent prediction.

Results

On hand manipulation trajectory prediction, ThinkJEPA outperforms strong VLM-only baselines and JEPA-predictor baselines.

---

*Auto-collected on 2026-03-25.*