WorldWeaver: When Multiple AIs Co-Create a Coherent Video World

This post is an in-depth Chinese-language walkthrough of the paper "Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers" (arXiv:2607.21594), which proposes WorldWeaver (W²)—a streaming multi-agent video diffusion model built around World State Registers.

The Problem: Consistency Without Communication

The author opens with an analogy: ten novelists take turns writing a novel, each seeing only what the previous one wrote. Over time, characters change appearance, dead characters reappear, and settings collapse—exactly the challenge of multi-agent world modeling: how can multiple independent generators maintain a consistent understanding of a shared world state without direct communication?

Existing autoregressive video diffusion models conditioned on observation history face two issues:

• The curse of observation history: in multi-view settings, per-agent histories overlap and diverge, making it hard to reconstruct a shared world state.
• State drift: errors accumulate over long generations, causing objects to shift, appearances to change, and physics to break.

Key Innovation: World State Registers

WorldWeaver introduces learnable tokens that act like a shared whiteboard, storing:

1. Shared world information — scene layout, static objects, environment state 2. Individual agent states — position, orientation, behavior, trajectories 3. Dynamic update logs — state-change history, interactions, causal tracking

The workflow: registers are initialized, read by each agent during generation, updated after every generated chunk, and propagated to the next step—enabling streaming generation of arbitrarily long videos.

Three Supervision Signals

• Individual agent state supervision (ground-truth positions/behaviors)
• Global state view supervision (bird's-eye views for scene layout)
• Scene text supervision (high-level semantic descriptions)

Architecture: Mixture-of-Transformers

World-state modeling (spatial reasoning, physics, causality) and visual frame generation (pixels, motion, appearance) are different tasks, so forcing shared weights causes them to pull against each other. WorldWeaver uses:

• A World State Transformer that updates registers
• A Visual Frame Transformer that generates frames conditioned on registers
• Loose coupling via registers, with no direct gradient flow from frames back to state modeling

The analogy: a film studio with separate writers' and cinematography departments communicating through a script.

Experiments: Two-Agent Minecraft

Minecraft was chosen for its rich interactivity, precisely representable discrete world states, and native multi-agent support. Two agents act in the same world, each with a first-person camera; the model must generate both viewpoint videos that remain logically consistent (agent A's appearance and position in agent B's view must match A's own view).

Findings:

• Logical consistency: significantly better cross-view consistency than baselines (object position, agent appearance, synchronized environment changes)
• Visual quality: no noticeable degradation despite the extra modeling component
• Long videos: baselines degrade quickly with length; WorldWeaver's registers anchor consistency much longer

Ablations confirm each piece matters: removing the registers collapses consistency; removing bird's-eye supervision hurts complex scenes; removing text supervision harms semantics; shared weights hurt both quality and state modeling.

Broader Significance and Limitations

The post connects world models to intelligence itself—prediction, counterfactual reasoning, and imagination all require internal world models—and sketches applications in autonomous driving simulation, VR, multi-robot collaboration, and game AI.

Honest limitations noted:

• Minecraft's discrete, blocky world is simpler than reality
• Only two agents were validated; state complexity may grow exponentially with agent count
• Heavy reliance on supervision signals that may be unavailable in the real world

Future directions include unsupervised/self-supervised state learning, hierarchical world models, probabilistic world states, and causal (not just descriptive) state representations.

Conclusion

WorldWeaver's core lesson: the key to multi-AI collaboration is not more complex instructions but a shared understanding space—a common, updatable representation of what the world looks like now, where each agent is, and what has changed. It represents a shift from "feed AI more data" to "give AI an understanding of world structure."

References

• Mo, S., et al. (2026). *Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers*. arXiv:2607.21594.
• Ha, D., & Schmidhuber, J. (2018). *World Models*. NeurIPS 2018.
• OpenAI (2024). *Sora: A text-to-video model*; Brooks, T., et al. (2024). *Video generation models as world simulators*.
• Girdhar, R., et al. (2023). *Emu Video*. arXiv:2311.10709.
• Vaswani, A., et al. (2017). *Attention is all you need*. NeurIPS 2017.