From Pixels to States: Can AI Video Generation Models Become the Next Game Engines?

This post is a detailed Chinese-language walkthrough of the paper "From Pixels to States: Rethinking Interactive World Models as Game Engines" by Zhen Li, Zian Meng, Shuwei Shi, Mingliang Zhai, Jiaming Tan, Chuanhao Li, and Kaipeng Zhang (arXiv:2607.14076). Below is an English structured summary of the discussion.

Key points

• Central question: Can video generation models (Sora, Veo, Gen-3, etc.) serve as next-generation game engines? The paper offers the first systematic academic assessment of this idea.
• Three unavoidable requirements for a genuine interactive game world:
1. Rule consistency — state changes (damaged enemies, broken objects) must persist; impossible if each frame is generated independently. 2. Long-horizon persistence — early-game choices must causally affect events hours later, far beyond current short-term temporal consistency. 3. Real-time generation — games need instant responses; current generative models take seconds to tens of seconds.
• The missing layer is state: Traditional engines follow an *action → explicit state update (by rules) → render observation* loop. Video models attempt a black-box shortcut *action → observation*, which sacrifices the state layer that guarantees all three requirements.

Four dimensions of current research

1. Player action control

• Direct conditioning: actions fed as input signals — simple but coarse control.
• Trajectory control: spatially precise (camera/character paths), but controls outcomes, not rules.
• State conditioning: explicit game states as generation conditions — closest to traditional engines and the paper's favored direction, though it requires costly state-observation annotated data.

2. Game state dynamics

• Implicit dynamics: fully data-driven; flexible but fragile on out-of-distribution cases.
• Explicit dynamics: hand-designed rules like traditional engines; reliable but rigid.
• Hybrid: explicit state guarantees rule consistency while a generative model handles rendering and detail — the paper's most promising direction.

3. State-observation persistence

• Approaches: sliding windows (short memory only), external memory modules (write/read still open problems), and state caching (save-game style snapshots).
• The paper distinguishes state persistence (the world's underlying truth, e.g., an NPC's death) from observation persistence (visual consistency, e.g., a wall seen from different angles) — two distinct problems.

4. Real-time interactive generation

• Distillation & compression, cascade generation (low-res first, then upsample), latent-space computation (e.g., VideoRAE-style), and pre-computation & caching.
• The paper's verdict is cautious: current techniques remain far from true real-time interactive generation, though multiple directions show promise.

The Black Myth: Wukong dataset

A major practical contribution: a large-scale dataset of 90+ hours of real gameplay with every frame aligned to player actions, game states, and visuals, plus structured semantic annotations. Rationale for choosing *Black Myth: Wukong* (Game Science's 3A action RPG):

• High visual complexity (environments, combat effects) as a stress test for generative models.
• Deep action mechanics (combos, dodges, blocks) for studying state-action dynamics.
• Distinct Eastern-mythology aesthetics complementing Western-dominated datasets.

It enables research on state-aware world models (bidirectional state↔image mapping), action-conditioned video generation, long-horizon consistency evaluation, and cross-scene generalization.

Deeper implications

• From generating video to simulating worlds: a video generator is a vending machine (independent input-output transactions); a world simulator is a sandbox where changes persist. The paper argues explicit state representation is a *mathematical necessity* — no state, no memory; no memory, no causality; no causality, no world.
• Two futures: Path A — AI-enhanced traditional engines (AI-generated assets, NPC behavior, level design, rendering), reliable and controllable in the short term; Path B — end-to-end neural engines, flexible but currently uncontrollable and unexplainable. The paper sees Path A as realistic now, with likely convergence later.
• Beyond games: world models are a core stepping stone toward AGI, and games are an ideal training ground — clear rules, unlimited interaction data, clear evaluation, and low failure cost.

Conclusion

The paper's value lies less in technical breakthroughs than in correctly framing the problem: video generation is powerful, but the gap to interactive worlds lies in the explicit state layer connecting actions, rules, and observations — and resources like the *Black Myth: Wukong* dataset are needed to bridge it.

Reference: Li, Z., et al. (2026). *From Pixels to States: Rethinking Interactive World Models as Game Engines.* arXiv:2607.14076.