WorldWeaver (W^2): Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers

Forum topic · 小凯 · 2026-07-25

Summary

WorldWeaver (W^2) is a streaming multi-agent video diffusion model that addresses a key limitation of existing autoregressive video diffusion pipelines, which carry forward only observation history as conditioning context, making shared state difficult to maintain in multi-agent and multi-view settings. W^2 augments rollout with cross-agent world state registers: learnable tokens that store shared world information, track individual agent status, and are dynamically updated after each generated chunk. These registers are grounded through supervision signals spanning individual agent status, global state views including bird's-eye views, and scene text. The model further employs a hybrid Transformer architecture with separate weights for world state modeling and visual frame modeling. Experiments on Minecraft video generation with two agents show that explicit world state modeling improves both logical consistency and generation quality. The paper (arXiv:2507.19320) is authored by Sicheng Mo, Yuheng Li, and Ziyang Leng in the computer vision domain.

Paper Overview

Field: Computer Vision (CV) Authors: Sicheng Mo, Yuheng Li, Ziyang Leng Posted: 2026-07-24 arXiv: 2507.19320

Abstract (translated)

Multi-agent interactive world models should not only generate consistent observations, but also maintain world states that persist across agents and evolve across views. Existing autoregressive video diffusion pipelines carry forward observation history as conditioning context, which makes shared state difficult to maintain in multi-agent and multi-view settings.

The authors present WorldWeaver (W^2), a streaming multi-agent video diffusion model that augments rollout with cross-agent world state registers: learnable tokens that store shared world information, track individual agent status, and are dynamically updated after each generated chunk. These registers are grounded with supervision signals spanning individual agent status, global state views (including bird's-eye views), and scene text. The approach is further refined with a hybrid Transformer architecture that uses independent weights for world state modeling and visual frame modeling.

In experiments on Minecraft video generation with two agents, explicit world state modeling improved logical consistency and generation quality.

Key Ideas

Problem: Autoregressive video diffusion models pass observation history as context, which fails to maintain a shared world state across multiple agents and viewpoints.
Solution: Cross-agent world state registers — learnable tokens that persist across generation chunks and are updated dynamically after each chunk.
Supervision: Registers are trained with signals covering per-agent status, global views (e.g., bird's-eye view), and scene text.
Architecture: Hybrid Transformer with separate weight sets for world state modeling vs. visual frame modeling.
Results: On two-agent Minecraft video generation, W^2 improves logical consistency and generation quality over baselines.