WorldWeaver: When Multiple AIs Co-Create a Coherent Video World

> "One person paints a picture, two people write a poem, three people weave a world."

This is a deep-dive interpretation of the paper "Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers" (arXiv:2607.21594), which proposes WorldWeaver (W2), a streaming multi-agent video diffusion model built around World State Registers.

Key points

The problem: consistency without communication

• The post opens with an allegory: ten writers take turns writing a novel, each seeing only the previous passage. Without a shared outline, characters change appearance, dead characters reappear, and settings get overturned. This mirrors the core challenge of multi-agent world modeling: keeping independent agents consistent about a shared world state.
• Existing autoregressive video diffusion models condition on observation histories, but in multi-agent settings these histories overlap and diverge across agents, making it hard to reconstruct a shared world state. Errors also accumulate (state drift) over long generations: objects shift, appearances change, physics get violated.

Core idea: World State Registers

• A set of learnable tokens acts as a shared "script" or whiteboard, storing:
1. Shared world information — scene layout, static objects, environment state 2. Individual agent status — each agent's position, orientation, behavior state, trajectories 3. Dynamic update log — state change history, interaction events, causal traces
• Dynamic update mechanism: registers are initialized, read during generation of each video chunk, updated after each chunk, and propagated to the next generation step.
• Three supervision signals train the registers:
1. Individual agent state supervision (ground-truth positions/orientations/actions) 2. Global state view supervision (e.g., bird's-eye views) 3. Scene text supervision (high-level semantic descriptions)

Architecture: Mixture-of-Transformers

• World-state modeling (spatial reasoning, physics, causality — a "planning" task) and visual frame modeling (pixels, texture, motion — a "rendering" task) conflict when forced to share weights.
• WorldWeaver therefore uses decoupled weights with information flow:
• A world-state transformer updates the registers from current state + agent actions/observations.
• A visual-frame transformer generates frames conditioned on the registers and previous frames.
• The two are "loosely coupled" through the registers; the analogy given is a film studio's screenwriting and cinematography departments communicating via the script.
• Streaming generation: long videos are generated chunk by chunk, with registers updated between chunks, allowing arbitrarily long videos in principle.

Experiments: two-agent Minecraft generation

• Minecraft was chosen for its rich physics, precise discrete world state, and native multi-agent support. Two agents act in one world, each with a first-person view; the model must generate both videos consistently (agent A must appear in agent B's view where A's own view says it is).
• Results:
• Significantly improved cross-view logical consistency vs. observation-history baselines (reduced object position drift, consistent agent appearance, synchronized environment changes).
• Visual generation quality is preserved — evidence the decoupled Mixture-of-Transformers design avoids task interference.
• As video length grows, baseline consistency degrades quickly while WorldWeaver degrades more slowly, thanks to register "anchoring."
• Ablations: removing the registers collapses consistency to near-baseline; removing bird's-eye-view supervision hurts complex scenes; removing text supervision harms semantic understanding; using shared weights instead of Mixture-of-Transformers degrades both quality and state modeling.

Broader significance

• World models relate to the hypothesis that intelligence rests on internal models enabling prediction, counterfactual reasoning, planning, and imagination.
• Multi-agent settings raise questions about subjective perspectives, emergent social phenomena, and the balance between consensus and useful divergence.
• Potential applications beyond Minecraft: autonomous-driving simulation, multi-user VR consistency, robot collaboration, and game NPC coordination.

Limitations and future directions

• Limitations: Minecraft is a discrete, simplified world; experiments cover only two agents (state complexity grows with agent count); training depends on explicit supervision signals.
• Future work: unsupervised/self-supervised state learning, hierarchical world models (objects → scenes → events), probabilistic world states for uncertainty, and causal (not just descriptive) world modeling — toward general world models, interactive world simulators, and shared human–AI world understanding.

Conclusion

The post's takeaway: the key to multi-agent collaboration is not more complex instructions, but a shared understanding space — a common representation of what the world currently looks like, where other agents are, and what has changed. WorldWeaver's registers, dynamic updates, and decoupled transformers offer an elegant engineering answer, marking a shift from "showing AI more data" to "giving AI an understanding of world structure."

References

• Mo, S., Li, Y., Leng, Z., Singh, K. K., & Zhou, B. (2026). Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers. *arXiv preprint* arXiv:2607.21594.
• Ha, D., & Schmidhuber, J. (2018). World Models. *NeurIPS 2018*.
• OpenAI. (2024). Sora: A text-to-video model. *OpenAI Technical Report*.
• Brooks, T., et al. (2024). Video generation models as world simulators. *OpenAI Research*.
• Girdhar, R., et al. (2023). Emu Video: Factorizing text-to-video generation by explicit image conditioning. *arXiv preprint* arXiv:2311.10709.
• Vaswani, A., et al. (2017). Attention is all you need. *NeurIPS 2017*.