Overview
AnchorWorld turns world models from passive predictors into editable interactive environments: your body movements drive the first-person camera, while user-placed anchor views define local world state in 3D space.
Why existing world models fall short
- PlayerOne (NeurIPS 2025) drives first-person views with full-body motion; CaM (SIGGRAPH Asia 2025) maintains consistency via history frames. But both leave the world passively determined by the initial frame — you cannot specify "there should be a flower here, and it is wilting."
- an RGB image, injected via in-context frame-dimension concatenation, distinguished by 3D RoPE
- a 3D pose, injected as broadcast pose embeddings added to visual tokens
- an evolution prompt (e.g., "this flower is slowly wilting"), injected via cross-attention with masks so each prompt only affects its own anchor and the video — enabling per-region dynamics without interference
- Ego Static: best on all metrics (PSNR 16.06, SSIM 0.578, ATE 0.112) vs PlayerOne and CaM variants.
- UE CineScene (anchors entirely outside initial view, out-of-distribution): still first on matched pixels and rotation accuracy, showing true spatial pose association rather than memorization.
- Ego Dynamic: text alignment 0.717 vs best baseline 0.449 — precise local dynamic control; global-text baselines fail to disambiguate regions.
- Out-of-sight evolution: with an anchor describing a person standing up from a sofa outside the initial view, the model correctly generates the temporal sequence once the camera turns — evidence of implicit 3D scene modeling, not image stitching.
- Removing third-person pretraining: RRE degrades 3.187 → 3.532
- Removing head pose: RRE 3.806
- Removing anchor 3D RoPE: matched pixels 4493 → 4395
- Progressive training beats joint training across stages
- Short horizons only (77 frames, 480p); no long-term memory or topological map
- Open-world generalization unverified; anchors require user-provided images
- Dynamics mostly limited to human activity; complex physics/fluid/weather not covered
- Li, Yu, et al. "AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customization." arXiv:2606.07326
- Project page: https://yuli0103.github.io/AnchorWorld/
- Base model: Wan2.2 TI2V 5B (Kuaishou Kling); data: Ego-Exo4D, LEMMA, MultiCamVideo; motion estimation: GVHMR
Technical approach
Projection-based action representation. Egocentric video lacks visible body signals, so AnchorWorld uses third-person video as auxiliary training data. 3D full-body motion (SMPL-X sequences, 22 joints) plus camera trajectories are projected into different 2D views. Pretraining on large third-person video (200k single-person clips + 101k UE multi-camera clips), then fine-tuning on egocentric paired data (Ego-Exo4D, LEMMA).
Spatial pose attention. Based on Wan2.2 TI2V 5B (a flow-matching DiT), motion tokens, camera tokens, and video tokens are concatenated along the spatial dimension into the same self-attention space — no extra parameters or adapters.
Anchor views. Each anchor contains:
Four-stage progressive training: (I) third-person motion pretraining → (II) egocentric adaptation → (III) static anchor customization → (IV) dynamic anchor evolution. Joint training of stages III & IV degrades static consistency, validating the staged design.