Marionette: Predicting Explicit World States for Interactive Game World Models (arXiv 2508.08542)

Forum topic · 小凯 · 2026-08-18

Summary

Marionette is a world model for interactive games with articulated characters that explicitly models an evolving 276-dimensional 3D world state instead of relying on latent pixel-space autoregression. A two-stage autoregressive dynamics model predicts multi-entity articulated skeletons, metric root trajectories, and rotations; a zero-parameter graphics bridge converts the predicted state into pose-controlled video with closed-form world-space geometry and occlusion; and a control-conditioned video diffusion model synthesizes realistic RGB observations. Experiments show the predicted state is directly controllable—forcing mismatched action streams shifts root-aligned joint error by 31% across 48 held-out clips—and that long-horizon failures can be fixed at the state level: two rule-based constraints (terrain collider and separation cap) reduce ground penetration by 66% without altering the observation model, while FVD of 831 matches recorded-pose rendering at 799.

Overview

Field: Computer Vision
Authors: Zian Meng, Zhen Li, Chuanhao Li
Published: 2026-08-17
arXiv: 2508.08542

Summary (translated)

Interactive game world models typically autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence. Over long horizons, errors in these latent world properties accumulate, making consistency and controllability fragile. Marionette instead explicitly models the evolving world state, delegates exact geometric computation to a fixed, zero-parameter renderer, and leaves the neural model to synthesize appearance.

The system has three components:

1. Dynamics model: A two-stage autoregressive model predicts an explicit, interpretable 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. 2. Graphics bridge: A zero-parameter renderer converts the predicted state into pose-controlled video, computing world-space geometry and occlusion in closed form. 3. Observation model: A control-conditioned video diffusion model synthesizes realistic RGB observations from the generated structured control.

Key experimental findings

Direct controllability: Forcing mismatched action streams changes root-aligned joint error by 31% across 48 held-out clips.
Long-horizon behavior is state-determined and state-fixable: Unconstrained, two generated characters drift up to 21.2 meters apart (recorded sessions stay around 5 meters), and a third of frames show ground penetration. Two explicit state-level rules—a terrain collider and a separation cap—reduce penetration by 66% while preserving character interaction, with no changes to the observation model.
No fidelity loss: Routing appearance through predicted states yields FVD of 831 versus 799 for recorded poses—a detectable gap the authors report as negligible.