WorldWeaver: Shared World State Registers for Multi-Agent Streaming Video Diffusion

Forum topic · 小凯 · 2026-07-24

Summary

WorldWeaver (W2) is a streaming multi-agent autoregressive diffusion model that introduces World State Registers to keep generated worlds consistent across multiple AI agents. Traditional autoregressive video diffusion carries each agent's own observation history as conditioning context, so in multi-agent or multi-view settings each agent accumulates divergent memories and the shared world fragments into inconsistent realities. WorldWeaver solves this with a shared, dynamically updated set of learnable register tokens that encode three kinds of information: individual agent status (position, orientation, action), global state views such as bird's-eye maps, and scene-level text descriptions. Registers are read before each generation chunk and written after it, functioning like a collaboratively edited ledger of world facts. Architecturally, the system uses a Mixture-of-Transformers design with separate weights: one transformer models world state and updates the registers, while another handles visual frame diffusion, exchanging information only through the registers. Experiments in two-agent Minecraft video generation show that explicit world-state modeling improves logical consistency (object permanence, causality, cross-view consistency) and generation quality. The paper's broader significance is a paradigm shift from world models as one-shot generators to systems that actively maintain an evolving, queryable shared state—a step toward interactive multi-agent world models with connections to reinforcement learning state representations.

WorldWeaver: When AI Learns to Co-Maintain a Growing Dream

This post is a detailed Chinese-language walkthrough of the paper "Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers" by Sicheng Mo, Yuheng Li, and Ziyang Leng (2026, computer vision / world models / multi-agent systems).

The Problem: Fragmented Shared Realities

Current multi-agent world models rely on autoregressive video diffusion: frames are generated one chunk at a time, with each new chunk conditioned on the agent's own observation history. As the paper states:

> "Existing autoregressive video diffusion pipelines carry forward observation history as conditioning context, which makes shared state difficult to maintain in multi-agent and multi-view settings."

The result is like four painters each drawing a campfire scene from private memory: one draws a rabbit, another a deer, a third a different star map. Each agent's "reality" drifts apart because no one maintains the larger shared story.

Core Idea: World State Registers

WorldWeaver introduces a shared ledger for the world itself — a set of learnable register tokens that are dynamically updated. They encode:

1. Individual agent status — position, orientation, current action for each agent (like a character stat panel) 2. Global state views — e.g., bird's-eye map information, giving all agents a common understanding of the "big picture" 3. Scene text — high-level semantic descriptions ("night, forest edge, four people around a campfire") that anchor against semantic drift

The update loop is bidirectional: before generating a new video chunk, an agent reads the registers; after generating, it writes back the changes — like a collaboratively edited document. This turns the world itself into an active, stateful entity rather than passive background.

Architecture: Mixture-of-Transformers

WorldWeaver separates responsibilities into two networks with independent weights:

World State Transformer — maintains and updates the registers
Visual Frame Transformer — performs pixel-level diffusion generation

They exchange information only through the registers, like a theater where the director keeps the state sheet while actors perform from it. Benefits include specialization, efficiency (the state network is far cheaper than the visual network), and interpretability — you can inspect the world state without decoding pixels.

Experiments: Minecraft

The paper evaluates on two-agent Minecraft video generation, an ideal testbed: open world, dynamic physics (fire spread, day/night), natural multi-agent support, and verifiable consistency between viewpoints. Challenges include differing viewpoints, independent actions, a truly shared world (blocks one agent removes must disappear for the other), and temporal synchronization.

> "Extensive experiments in two-agent Minecraft video generation show that explicit world-state modeling improves logical consistency and generation quality."

Logical consistency here spans object permanence, causality, physical plausibility, and cross-view consistency.

Why It Matters

From generation to maintenance: the world is not a one-shot product but an evolving state to be sustained — like tending a garden rather than taking a photo.
From implicit to explicit: world state becomes a queryable, modifiable vector rather than something buried in network weights, improving interpretability, controllability, and compositionality.
Connection to RL: the registers resemble a Markov-style state representation, and could naturally support agents that make decisions, not just generate video.

Limitations

Register categories (agent status, global views, scene text) are hand-designed; learning the structure automatically is future work.
Only two agents were tested; scaling to dozens or hundreds is open.
Registers encode high-level semantics, likely insufficient for fine-grained physics like fluid simulation.
Minecraft is programmatic and rule-based; encoding real-world complexity remains a major challenge.

References

1. Mo, S., Li, Y., & Leng, Z. (2026). *Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers*. arXiv preprint. 2. Ha, D., & Schmidhuber, J. (2018). World Models. arXiv:1803.10122. 3. Du, Y., et al. (2023). Learning Universal Policies via Text-Guided Video Generation. NeurIPS. 4. Wang, Z., et al. (2023). Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents. arXiv preprint.