Pelican-Sim 1.0: A Dream-Like World Model Simulator with 28-Dimensional Actions and 4-Step Generation

Forum topic · QianXun · 2026-10-02

Summary

Pelican-Sim 1.0, released by the Beijing Innovation Center of Humanoid Robotics (X-Humanoid) in September 2026 (arXiv:2609.12036), is a general video world-model simulator for embodied AI. It uses a unified 28-dimensional action representation (14 per arm: 7 joint angles, gripper open/close, and 6 dexterous-hand values, with zero-padding for missing degrees of freedom) plus a projected skeleton video rendered from URDF forward kinematics. Both condition types are injected alternately into a 28-layer Video DiT, with dense FFNs replaced by a sparse MoE (1 shared expert, 8 routed experts, top-2) that cuts FVD from 17.380 to 10.850. Distribution-matching distillation compresses 35 denoising steps to 4, yielding a 5.67x speedup (2.2 s per 21-frame trajectory). Trained on about 1 million trajectories (8,000 hours) from three real and four simulated sources, it reaches PSNR 22.276 on AgiBotWorld Beta and 30.383 on RoboTwin. Four downstream uses are demonstrated: data generation boosting a pi0.5 policy from 70% to 93% success, policy evaluation and ranking with 91.4% accuracy on generated video, best-of-10 action selection reaching 63.8%, and closed-loop GRPO training lifting OpenVLA-OFT from 62.7% to 75.4%, still 8.4 points below an oracle simulator. Limitations include unreported parameter counts and data-mix ratios.

Key points

Pelican-Sim 1.0 (arXiv:2609.12036), from the Beijing Innovation Center of Humanoid Robotics (X-Humanoid), positions a generative world model as a repeatable "dream" environment where robot policies can be evaluated, ranked, selected, and improved without touching real hardware. The source post analyzes the paper's architecture, ablations, efficiency, and four downstream applications.

Unified action representation

28-dim actions: 14 per side — 7 arm joint angles, 1 gripper open/close, 6 dexterous-hand joint values; missing dimensions zero-padded (e.g., the 7th joint of 6-axis arms).
Action video: the numeric vector is projected via URDF forward kinematics and camera calibration into a rasterized full-body skeleton video aligned with RGB; gripper links are colored by open / transitional / closed state.
Both are injected into a single 28-layer Video DiT: numeric actions via scale-and-shift modulation on odd layers (1, 3, … 27), action video via Context Block residuals on even layers (0, 2, … 26). Alternating injection loses only 0.075 dB PSNR versus injecting both at every block (AgiBotWorld Beta).
Conditioning ablation (PSNR): none 17.684, numeric only 19.238, video only 21.276, both 22.276.

Sparse MoE and 4-step sampling

Dense FFNs in all 28 blocks are replaced by MoE: 1 shared expert, 8 routed experts, top-2 per token, sigmoid token-choice routing with selection bias (router ramped over 2,000 steps). FVD drops from 17.380 to 10.850 at identical training cost.
Two-stage speedup: causal adaptation (block-causal attention → 35-step causal teacher), then DMD distillation to a 4-step student with a few teacher-forced updates for action controllability. On a 21-frame baseline: 2.2 s per trajectory, 5.67x speedup, with KV-cached rolling context at inference.

Throughput and training data

| Method | s/trajectory | trajectories/hour | |---|---|---| | Human collection | 60 | 60 | | Pelican-Sim 1.0 (vs. human) | 24 | 150 | | RoboTwin simulator | 55 | 65.5 | | Pelican-Sim 1.0 (vs. simulator) | 19 | 189.5 |

Training: ~1M trajectories, 8,000 hours; 3 real sources (AgiBotWorld Beta, RealSource World, RoboMIND) and 4 simulated sources (LIBERO, ManiSkill2, RoboTwin, RoboCasa). Source mix ratios are not reported.

Image quality (within-source held-out splits, 90/5/5)

| Dataset | Best baseline PSNR | Pelican-Sim | Gain | |---|---|---|---| | AgiBotWorld Beta | 17.640 | 22.276 | +4.636 | | RoboMIND | 21.770 | 23.850 | +2.080 | | RoboTwin | 20.040 | 30.383 | +10.343 |

FVD on RoboTwin falls from 26.600 to 4.980. Gains hold only on within-distribution test sets; the paper's own text notes "within-source partitions rather than entirely unseen datasets," and OOD rows in Table 1 use each method's own settings.

Four downstream applications

1. Data generation: from 50 clean demos per task, GPT-Image-2 creates 10 initial-frame variants, each rolled out — 500 synthetic trajectories. Testing π0.5 on 5 RoboTwin tasks (200 unseen seeds each): 10 demos → 28.5% → 64.5%; 30 demos → 57.0% → 87.0%; 50 demos → 70.0% → 93.0%. 2. Policy evaluation/ranking: a Qwen3-VL-2B-Instruct + LoRA evaluator (16 frames per query) scores final success, 5-level progress, and frame validity. Accuracy: 92.6% on 1,000 real RoboTwin videos vs. 91.4% on 700 generated ones (1.2 pp gap — a gap in judging depicted outcomes, not a fidelity measurement). Policy ranking with 200 adapted rollouts matches fully (Spearman 1.000). 3. Action selection: best-of-10 candidates scored by predicted futures reaches 63.8% vs. 43.2% single-sample, within 4.3 pp of the 68.1% oracle; gains flatten after 8 candidates. 4. Policy improvement (GRPO): RL purely inside the simulator with VLM scores as reward lifts OpenVLA-OFT (SFT) from 62.7% to 75.4% (+12.7 pp absolute) — still 8.4 pp below the RoboTwin-oracle GRPO result of 83.8%.

Stated limitations

The URDF-rendered action video is nominal kinematic guidance, not a hard constraint on generated RGB motion.
Policy evaluation uses a matched-action protocol (fixed trajectory replay), not closed-loop execution under generated observations.
Mixed-domain ablations confound source diversity with total data volume.
Some adapted EWMBench sub-metrics remain below the best baselines; total parameters and data-mix ratios are unreported.

Analysis

The post highlights that the paper treats the evaluation chain as a first-class citizen: the VLM evaluator is calibrated on real simulator video before scoring generated rollouts, grounding the four applications. Two open concerns: the 8.4 pp oracle gap (whether from VLM scoring granularity or weak contact-dynamics modeling is unexamined) and the transparency of unreported data-mix proportions and parameter counts.

Source: arXiv:2609.12036v1 (Tables 1–14 and appendices); a Tencent News Chinese explainer (2026-10-02) used only as cross-reference, with all figures taken from the paper.