From Pixels to States: Rethinking Interactive World Models as Game Engines

Overview

This is a Chinese forum post (zhichai.net) presenting an in-depth reading of the paper "From Pixels to States: Rethinking Interactive World Models as Game Engines" by Zhen Li, Zian Meng, Shuwei Shi, Mingliang Zhai, Jiaming Tan, Chuanhao Li, and Kaipeng Zhang (arXiv:2607.14076, computer vision / interactive world models / game AI).

Key points

The core question

• Framed with a narrative from John Carmack's Wolfenstein 3D (1992) through Minecraft, Zelda, GTA, and Black Myth: Wukong, the post asks: can video generation models (Sora, Veo, Gen-3) become the next game engines?
• The paper is described as the first systematic academic examination of this question — a navigation map showing where the field stands and what obstacles lie ahead.

Three non-negotiable conditions for a true interactive world

1. Rule consistency — consequences persist (a broken vase stays broken); traditional engines achieve this via an explicit game state data structure, while frame-by-frame video models lack such memory. 2. Long-horizon persistence — early choices affect the story ten hours later; current video models only ensure short-term temporal coherence. 3. Real-time generation — players cannot wait seconds per frame; engines render from known state instead of generating pixels from scratch.

The action-state-observation loop

• Traditional engines: player action → update state (by rules) → render observation → ...
• Video generation models skip the state layer: player action → [black-box model] → observation — the missing state layer is the foundation of all three conditions.

Four-dimension survey of current research

• Player action control: direct conditioning, trajectory control (camera/position), and state conditioning (closest to traditional engines but needs costly state-observation annotated data).
• Game state dynamics: implicit (data-driven, flexible but fragile), explicit (rule-based, reliable but rigid), and hybrid — the most promising direction per the paper (explicit state for rules + generative model for rendering/detail).
• State-observation persistence: sliding windows, external memory modules, state caching; the paper distinguishes state persistence (world truth consistency) from observation persistence (visual consistency).
• Real-time interactive generation: distillation/compression, cascade generation, latent-space manipulation (e.g., VideoRAE-style), pre-computation and caching. The paper's cautious verdict: a significant gap remains before true real-time interaction.

The Black Myth: Wukong dataset

• Over 90 hours of real gameplay, with every frame precisely aligned to player actions, game states, and visuals, plus structured semantic annotations.
• Chosen for visual complexity, deep combat mechanics, distinctive Eastern-myth aesthetic, and community reach.
• Enables research on state-aware world models, action-conditioned video generation, long-horizon consistency evaluation, and cross-scene generalization.

Deeper reflections

• A video-generating AI is like a vending machine (independent transactions, no memory); a world-simulating AI is like a sandbox (actions permanently change state). The paper's central thesis: explicit state representation is a mathematical necessity, not a design preference.
• Two futures: Path A — AI-enhanced traditional engines (AI-generated assets, NPC behavior, level design, rendering); reliable and controllable. Path B — end-to-end AI engines; flexible but currently unreliable and unexplainable. Short-term: Path A; long-term: likely convergence.
• World models are a core path toward AGI; games are an ideal low-cost testbed with clear rules, unlimited interaction data, and evaluation criteria, with transfer potential to robotics, autonomous driving, and simulation.

Conclusion

• The paper offers a clear problem framework, a map of the current landscape, and a valuable dataset rather than a final answer. Its significance lies in correctly defining the problem: the gap between video generation and interactive worlds is fundamentally about state.

Reference

Li, Z., Meng, Z., Shi, S., Zhai, M., Tan, J., Li, C., & Zhang, K. (2026). From Pixels to States: Rethinking Interactive World Models as Game Engines. *arXiv preprint* arXiv:2607.14076.