From Pixels to States: A Survey on Interactive World Models as Game Engines

Forum topic · 小凯 · 2026-07-17

Summary

This arXiv paper (2607.14076) surveys interactive world models from the perspective of conventional game engines' action-state-observation loop. The authors—Zhen Li, Zian Meng, and colleagues from a CV research team—argue that while video generative models are increasingly viewed as next-generation game engines, truly interactive worlds require rule-consistent interaction outcomes, long-horizon persistence, and real-time generation. They organize the field along four dimensions: player action control, game state dynamics, state-observation persistence, and real-time interactive generation, grouping existing methods into representative families and analyzing their strengths and trade-offs. As a supplementary contribution, they release a scalable data engine built on the game Black Myth: Wukong, collecting over 90 hours of gameplay with frame-aligned player actions, real game states, visual observations, and structural/semantic annotations to support state-aware world modeling.

Paper Overview

Research Area: Computer Vision (CV) Authors: Zhen Li, Zian Meng, Shuwei Shi, Mingliang Zhai, Jiaming Tan, Chuanhao Li, Kaipeng Zhang arXiv: 2607.14076

Abstract

Building interactive worlds that respond coherently to player actions has long been a shared goal of computer graphics, games, and artificial intelligence. Recent video generative models provide a data-driven route toward this goal by predicting future observations conditioned on user actions, and are increasingly regarded as potential next-generation game engines.

Realizing a genuinely interactive game world, however, requires:

Interaction outcomes that follow rules over evolving game conditions
Consequences that persist over long horizons
A generation loop that operates in real time

Conventional game engines realize these properties through a recurrent action–state–observation loop, in which player actions update an explicit game state according to predefined rules, and observations are rendered from the resulting state.

Four Dimensions of Analysis

Using this loop as an organizing lens, the survey examines interactive game world modeling along four dimensions:

1. Player action control 2. Game state dynamics 3. State–observation persistence 4. Real-time interactive generation

For each dimension, the authors group existing methods into representative families based on the capabilities required for interactive game worlds, and discuss the strengths and trade-offs of each family.

Supplementary Contribution: Data Engine

As complementary analysis, the paper presents a scalable data engine built on *Black Myth: Wukong*, collecting over 90 hours of gameplay including:

Frame-aligned player actions
Real game states and visual observations
Structural and semantic annotations

This dataset serves as a resource for state-aware game world modeling.

The authors hope the survey provides a clear picture of the current state of the field and catalyzes progress toward truly interactive game worlds.

--- *Auto-collected on 2026-07-17*