WorldWeaver: Teaching AI Agents to Co-Maintain a Shared, Growing World State

Forum topic · 小凯 · 2026-07-24

Summary

WorldWeaver (W²) is a streaming multi-agent autoregressive diffusion model with World State Registers, proposed by Sicheng Mo, Yuheng Li, and Ziyang Leng. It tackles a core problem in multi-agent world models: when each agent autoregressively generates video from its own observation history, their shared world drifts out of sync, producing inconsistent scenes across views and agents. WorldWeaver introduces a shared, explicitly maintained 'ledger' for the world — a set of learnable register tokens that encode shared world information (time, weather, environment), per-agent status (position, orientation, action), and high-level scene text. Registers are dynamically updated: agents read the current world state before generating each video chunk and write back changes afterward. Architecturally, it uses a Mixture-of-Transformers design with separate weights: one Transformer models world state, the other generates visual frames via diffusion, exchanging information through the registers. Experiments in two-agent Minecraft video generation show that explicit world-state modeling improves logical consistency (object permanence, causality, cross-view consistency) and generation quality. The paper represents a shift from implicit to explicit world state, toward interactive, multi-agent world models, though questions remain about scaling to many agents, register design, and real-world complexity.

Paper: *Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers* Authors: Sicheng Mo, Yuheng Li, Ziyang Leng (arXiv preprint, 2026) Area: Computer vision / World models / Multi-agent systems

Key points

The problem: In existing autoregressive video diffusion pipelines, each agent carries forward only its own observation history as conditioning context. In multi-agent and multi-view settings, this causes shared state to drift — agents generate mutually inconsistent worlds (different timelines, skies, object placements). As the paper states: "Existing autoregressive video diffusion pipelines carry forward observation history as conditioning context, which makes shared state difficult to maintain in multi-agent and multi-view settings."
Core idea — World State Registers: Instead of letting each agent maintain private memory, WorldWeaver maintains a shared, dynamic ledger for the world itself: a set of learnable tokens that store shared world information (time, weather, environment), track individual agent status (position, orientation, current action), and hold high-level scene text descriptions. Scene text acts as a semantic anchor against visual drift.
Read/write updates: Before generating a new video chunk, an agent reads the current register state; after generating, it writes back the changes it introduced. The registers are thus continuously updated, making the world an actively maintained, stateful entity rather than a passive backdrop.
Architecture — Mixture-of-Transformers: WorldWeaver separates responsibilities into two networks with independent weights: a world-state Transformer that updates the registers, and a visual-frame Transformer that performs diffusion-based pixel generation conditioned on the registers. Benefits include specialization, efficiency, and interpretability of the world state without decoding pixels.
Three supervision signals: The registers are trained with (1) individual agent status, (2) global state views (bird's-eye-view map information), and (3) scene text descriptions.

Evaluation

Experiments use two-agent Minecraft video generation, chosen because Minecraft offers an open world, dynamic physical rules (fire spread, day/night cycles), natural multi-agent support, rich semantics, and verifiable consistency (one can check whether what agent A sees matches agent B's view). Agents occupy different positions, act independently, and their videos must be temporally synchronized and reflect each other's changes (e.g., a block removed by agent A must appear removed in agent B's view).

The paper reports:

> "Extensive experiments in two-agent Minecraft video generation show that explicit world-state modeling improves logical consistency and generation quality."

Logical consistency here covers object permanence, causality, physical plausibility, and cross-view consistency. The learned registers encode time progression, spatial relations, object interactions, and role differentiation between agents.

Significance and limitations

WorldWeaver reframes world models from "generators of video" to "maintainers of a coherent, queryable world state" — a shift from implicit to explicit state representation, analogous to moving from taking photos to tending a garden. It connects naturally to reinforcement learning, where register-style state representations provide the information needed for decision-making, and points toward interactive worlds where multiple AI agents share and co-evolve one reality.

Stated/identified limitations:

Register categories are human-designed (agent status, global views, scene text); automatic discovery of register structure is future work.
Only validated with two agents; scaling to dozens or hundreds of agents is open.
Registers encode high-level semantics, which may be too coarse for fine-grained physical simulation.
Minecraft is programmatic and rule-based; encoding real-world complexity (light transport, sound, human intent) remains a major challenge.

References

1. Mo, S., Li, Y., & Leng, Z. (2026). *Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers*. arXiv preprint. 2. Ha, D., & Schmidhuber, J. (2018). World Models. arXiv:1803.10122. 3. Du, Y., et al. (2023). Learning Universal Policies via Text-Guided Video Generation. NeurIPS. 4. Wang, Z., et al. (2023). Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents. arXiv preprint.