Marionette: Predicting World States, Rendering Geometry, Painting Appearance for Interactive Game World Models

Forum topic · 小凯 · 2026-08-18

Summary

Marionette (arXiv 2508.08542, Zian Meng, Zhen Li, Chuanhao Li) is a world model for interactive games with articulated characters that separates world-state reasoning from appearance synthesis. Instead of autoregressively generating pixels or latents, it explicitly predicts a 276-dimensional 3D world state containing multi-entity articulated skeletons, metric root trajectories, and rotations via a two-stage autoregressive dynamics model. A zero-parameter graphics bridge converts predicted states into pose-controlled video, computing world-space geometry and occlusion in closed form. Finally, a control-conditioned video diffusion observation model synthesizes realistic RGB frames from the structured control. Experiments show the predicted state is directly controllable: forcing mismatched action streams shifts root-aligned joint error by 31% on 48 held-out clips. Long-horizon behavior is state-determined and repairable at the state level: unguided sessions drift to 21.2 meters apart with one-third of frames showing ground penetration, while two state-level rules (terrain collider and separation cap) reduce penetration by 66% without touching the observation model. Routing appearance through predicted state preserves fidelity, with FVD of 831 versus 799 for recorded poses.

Paper Overview

Field: Computer Vision (CV) Authors: Zian Meng, Zhen Li, Chuanhao Li arXiv: 2508.08542

Abstract

Interactive game world models typically autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence. Over long horizons, errors in these latent world properties accumulate, making consistency and controllability fragile.

The authors instead explicitly model the evolving world state, delegate exact geometric computation to a fixed, zero-parameter renderer, and leave the neural model to synthesize appearance. They instantiate this idea as Marionette, a world model for interactive games with articulated characters, consisting of three components:

1. Dynamics model: a two-stage autoregressive model predicts an explicit, interpretable 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. 2. Graphics bridge: a zero-parameter renderer converts the predicted state into pose-controlled video, computing world-space geometry and occlusion in closed form. 3. Observation model: a control-conditioned video diffusion model synthesizes realistic RGB observations from the generated structured control.

Key Findings

Direct controllability of the predicted state: forcing mismatched action streams changes root-aligned joint error by 31% across 48 held-out clips.
State-determined long-horizon behavior, repairable at the state level: in free-running mode, two generated characters drift up to 21.2 meters apart (recorded sessions stay around 5 meters), and one-third of frames show ground penetration. Two simple state-level rules—a terrain collider and a separation cap—reduce penetration by 66% while preserving character interaction, without any change to the observation model.
No fidelity sacrifice: routing appearance through the predicted state yields an FVD of 831 versus 799 with recorded poses—no detectable degradation.