Programmable World Model: Decoupling World State from Generative Rendering

Forum topic · 小凯 · 2026-09-11

Summary

Programmable World Model (arXiv 2609.10540) is a framework that separates world-state evolution from visual observation generation in video world models. An agent translates natural-language instructions into executable programs defining entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes. To bridge world state and visual generation, the method introduces state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation, which, together with target camera trajectories, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model acting as a generative renderer. This design enables playable games with predefined mechanics, per-entity control, and persistent world state throughout gameplay. The authors also present CombatStateBench, a benchmark for evaluating programmable world models, on which the approach achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation.

Overview

Field: Computer Vision (CV) Authors: Zheng-Hui Huang, Guixu Lin, Jiacheng Lin, Yi-Chuan Huang, Ruihan Yu, Muyao Niu, Siqi Yang, Yu-Lun Liu, Yung-Yu Chuang, Kaipeng Zhang, Zhixiang Wang arXiv: 2609.10540

Abstract (translation)

Recent video world models generate increasingly realistic and interactive visual experiences, yet lack reliable mechanisms for maintaining persistent world state and enforcing programmable rules over extended interactions. We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation. An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules, enabling direct control over individual entities and their interactions. A lightweight engine executes these programs to update and maintain an explicit, persistent global world state, including off-screen entities and non-visual attributes.

To connect world state with visual generation, we introduce state-augmented 3D oriented bounding boxes (OBBs) as an intermediate representation. This representation, together with the target camera trajectory, is deterministically compiled into pixel-aligned spatiotemporal conditioning signals for a pretrained video model serving as the generative renderer. This design allows users to create playable games with predefined mechanics, direct control over individual entities, and persistent world state throughout gameplay.

We further introduce CombatStateBench, a benchmark for evaluating programmable world models. On CombatStateBench, our method achieves 94% Count Accuracy and 98% State Accuracy, substantially outperforming existing interactive video world models while supporting coherent long-horizon generation. These results demonstrate the effectiveness of separating explicit state evolution from generative rendering for building persistent, programmable worlds.

Key contributions

Decoupled architecture: world-state evolution is separated from visual observation generation.
Natural language to executable programs: an agent compiles instructions into programs specifying entity states and transition rules.
Persistent global world state: maintained by a lightweight engine, including off-screen entities and non-visual attributes.
State-augmented 3D OBBs: deterministic compilation into pixel-aligned spatiotemporal conditioning for a pretrained video renderer.
CombatStateBench: 94% Count Accuracy and 98% State Accuracy, outperforming prior interactive video world models.