Paper Overview
- Field: NLP / world models
- Authors: Haichao Zhang, Yijiang Li, Shwai He, Tushar Nagarajan, Mingfei Chen, Jianglin Lu, Ang Li, Yun Fu
- Published: 2026-03-23
- arXiv: 2603.22281
- A dense JEPA branch for fine-grained motion and interaction cues.
- A VLM thinking branch operating on uniformly sampled frames with a larger temporal span, providing rich knowledge-based guidance.
- arXiv: https://arxiv.org/abs/2603.22281
Summary
Recent progress in latent world models (e.g., V-JEPA2) has shown promising capability in forecasting future world states from video observations. Nevertheless, dense prediction from a short observation window limits temporal context and can bias predictors toward local, low-level extrapolation, making it difficult to capture long-horizon semantics and reducing downstream utility.
Vision-language models (VLMs), in contrast, provide strong semantic grounding and general knowledge by reasoning over uniformly sampled frames, but they are not ideal as standalone dense predictors due to compute-driven sparse sampling, a language-output bottleneck that compresses fine-grained interaction states into text-oriented representations, and a data-regime mismatch when adapting to small action-conditioned datasets.
Proposed Approach
The authors propose a VLM-guided JEPA-style latent world modeling framework that combines dense frame dynamics modeling with long-horizon semantic guidance through dual temporal pathways:
To effectively transfer the VLM's progressive reasoning signals, the paper introduces a hierarchical pyramid representation extraction module that aggregates multi-layer VLM representations into guidance features compatible with latent prediction.
Results
On hand manipulation trajectory prediction experiments, the method outperforms both strong VLM-only baselines and JEPA-predictor baselines.