AnchorWorld: Editable Egocentric World Simulation with Anchor Views and Full-Body Motion Control

Forum topic · 小凯 · 2026-06-09

Summary

AnchorWorld is a joint research effort from Tsinghua, HUST, HKUST, Wuhan University, and Kuaishou Kling that turns world models into editable, interactive environments. Built on Wan2.2 TI2V 5B, the system generates first-person egocentric video driven by full-body motion (SMPL-X sequences) while letting users place anchor views in 3D space. Each anchor combines an RGB image, a 3D pose, and an evolution prompt, enabling localized scene customization such as a flower that withers over time even when out of view. A key innovation is a projection-based action representation trained on large-scale third-person video (200K single-person clips plus 101K Unreal Engine multi-camera sequences) before fine-tuning on egocentric pairs from Ego-Exo4D and LEMMA, addressing the sparse motion supervision inherent to head-mounted cameras. Spatially concatenated action, camera, and video tokens share self-attention, and a four-stage progressive training schedule outperforms joint training in ablations. On static, dynamic, and out-of-distribution UE benchmarks, AnchorWorld beats PlayerOne (NeurIPS 2025) and CaM (SIGGRAPH Asia 2025) on scene consistency, motion accuracy, and text alignment (0.717 vs 0.449). Limitations include short 77-frame horizons, limited open-world generalization, and narrow dynamic content coverage.

AnchorWorld: Your Body Is the Controller, Anchors Are the World Editor

> Structured translation of a Chinese forum post by 小凯 (zhichai.net, 2026-06-09), analyzing the paper "AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customization" (arXiv:2606.07326).

Key points

Problem: Existing world models (PlayerOne, NeurIPS 2025; CaM, SIGGRAPH Asia 2025) can continue a scene but cannot *edit* it. The initial frame determines everything; users cannot specify "there should be a flower here, and it is wilting."
Solution: AnchorWorld combines (1) full-body motion-driven first-person view control and (2) user-placed anchor views in 3D space that define local world state.
Mixed-view training: Egocentric video lacks visible body, making motion supervision sparse. AnchorWorld pretrains on third-person video, then adapts to head-mounted views via a projection-based action representation that maps 3D motion into either camera type.
Anchors: Each anchor = RGB image + 3D pose + evolution prompt, injected respectively via in-context frame concatenation (with 3D RoPE to distinguish anchors), pose embeddings added to visual tokens, and masked cross-attention so prompts affect only their own anchor region.
Base model: Wan2.2 TI2V 5B (Kuaishou Kling), a flow-matching DiT. Motion (SMPL-X, 22 joints, no hands) and camera tokens are concatenated with video tokens along the spatial dimension and share self-attention; the generative backbone itself is unmodified.
Four-stage progressive training: third-person motion pretraining → egocentric adaptation → static anchor customization → dynamic anchor evolution. Ablations show progressive training beats joint training across metrics.

Technical details

Motion encoder projects SMPL-X sequence M ∈ ℝ^(f×k×6) to z_m; camera encoder projects pose sequence C to z_c; tokens concatenated as T = [z_v; z_m; z_c] ∈ ℝ^(f'×(h·w+k+1)×d), then truncated after generation.
Anchor image latents z_s concatenate along the frame axis: T_total = [z_v; z_s].
Evolution prompts use a strict attention mask: prompt j's keys are visible only to video tokens and anchor-j image tokens, preventing cross-anchor interference.
Training data: Ego-Exo4D, LEMMA, MultiCamVideo, 200K internal single-person clips, 101K UE multi-camera clips; poses estimated with GVHMR in a unified 3D coordinate system.

Results

| Setting | Highlights | |---------|-----------| | Ego Static | Best on all metrics: PSNR 16.06, SSIM 0.578, LPIPS 0.470, ATE 0.112, RRE 3.145, CLIP-V 0.885 | | UE CineScene (anchors outside initial view) | Matched pixels 4555.1K and RRE 1.656 — best, showing true spatial generalization, not initial-frame memorization | | Ego Dynamic | Text alignment 0.717 vs best baseline 0.449; PSNR 16.37; ATE 0.048 |

Key ablations: removing third-person pretraining raises RRE 3.187→3.532; dropping head pose raises RRE to 3.806; removing anchor 3D RoPE cuts matched pixels 4493→4395; joint Stage III/IV training degrades both static and dynamic metrics.

Qualitative tests confirm out-of-sight evolution (model generates a person standing up from a sofa outside the initial view, in correct temporal order) and spatial pose awareness (flipping poses/anchor poses yields sensible retrieval vs. plausible novel content), indicating an implicit 3D scene representation rather than 2D collage.

Limitations

Short horizon only (77 frames, 480p); no long-term memory or topological map for extended exploration.
Open-world generalization untested beyond domain shifts; anchors require user-provided images.
Dynamic content limited mostly to human activity; complex physics, fluids, and weather are not covered.

Significance

AnchorWorld shifts world models from passive predictors to editable interactive environments. For VR/AR, designers can define scenes via anchors and users explore with their bodies; for embodied AI, simulation states can be precisely configured per task.

References

Li, Yu, et al. "AnchorWorld: Embodied Egocentric World Simulation with View-based Evolution Customization." arXiv:2606.07326
Project page: https://yuli0103.github.io/AnchorWorld/
Baselines: PlayerOne (NeurIPS 2025), CaM (SIGGRAPH Asia 2025)