WOVEN: Weaving Visual World Modeling into Multimodal LLMs

Forum topic · 小凯 · 2026-10-11

Summary

WOVEN is a training source and benchmark for visual transition reasoning in multimodal large language models (MLLMs), introduced to address their systematic weaknesses in spatial, embodied, physical, and temporal reasoning. The dataset contains 36,076 realistic rollouts from video-pretrained generative models, organized by scene (20 types), action (5 types), and reasoning type (8 types). Evaluation of 38 frontier MLLMs, including GPT-5.4 and Qwen3-VL-235B-A22B, shows even the strongest models fall far below human performance, with failures recurring across model families and persisting at scale. Training MLLMs on ~2,000-example subsets each improved 22 of 26 external benchmarks by up to 27.3 percentage points, and WOVEN data could replace 30-50% of task-specific training data while maintaining accuracy. The study yields a training recipe: select supervision by reasoning operation rather than action, scene, or domain, and prefer larger visual state changes for robustness. arXiv: https://arxiv.org/abs/2610.12417

Paper Overview

Field: NLP Authors: Zheyu Fan, Yue Zhang, Mingkai Deng, Kangrui Wang, Qineng Wang, Canyu Chen, Jie Hao, Xing Fan, Chenlei Guo, Eric P. Xing, Mohit Bansal, Manling Li Published: 2026-10-08 arXiv: 2610.12417

Abstract

Multimodal large language models (MLLMs) struggle with spatial, embodied, physical, and temporal reasoning. The authors hypothesize that these failures reflect a shared deficit in visual transition reasoning, and test whether this capability can serve as a shared training primitive — one that different models can learn from different supervision sources and reuse across different tasks — with a systematic training recipe.

Existing benchmarks document these deficits separately but do not support controlled comparisons across scenes, actions, and reasoning operations. WOVEN fills this gap: a training source and benchmark for visual transition reasoning that organizes transition supervision by scene, action, and reasoning type, using diverse, realistic rollouts from video-pretrained generative models.

Key Findings

Dataset: 36,076 examples covering 20 scene types, 5 action types, and 8 reasoning types.
Benchmarking: 38 frontier MLLMs (e.g., GPT-5.4 and Qwen3-VL-235B-A22B) evaluated; even the strongest models fall far below human performance, with failures recurring across model families and persisting at scale.
Transferability: Training subsets of ~2,000 examples each collectively improved 22 of 26 external benchmarks by up to 27.3 percentage points.
Efficiency: WOVEN data can substitute 30-50% of task-specific training data while maintaining comparable accuracy.
Training recipe: Supervision should be selected by reasoning operation rather than action, scene, or domain; larger changes in visual state are preferred for robustness. The recipe was prospectively validated on held-out benchmarks.

The work establishes visual transition reasoning as a reusable foundation for systematically training visual world models in MLLMs.

--- *Auto-collected on 2026-10-11*