World-VLA-Loop: Closing the Loop Between Video World Models and VLA Policies
> Paper: *Closed-Loop Learning of Video World Model and VLA Policy* > Authors: Xiaokang Liu, Zechen Bai, Hai Ci, Kevin Yuchen Ma, Mike Zheng Shou (Show Lab, National University of Singapore) > arXiv: 2602.06508v1 [cs.RO] | Project page: https://showlab.github.io/World-VLA-Loop/
The Problem: Video World Models Hallucinate Success
Recent robot world models are built on video diffusion transformers: given history states and action sequences, they predict future visual observations. This promises virtual training environments without touching real hardware. However, they suffer from poor action-following precision. Models like Cosmos-Predict 2 can generate photorealistic video, but when fed an erroneous action, they often still predict a successful outcome — a phenomenon the paper calls action hallucination. The model relies on visual priors rather than simulating the underlying physical dynamics.
This is disastrous for reinforcement learning: the agent takes a wrong action, the world model claims success, RL assigns positive reward, and the resulting policy fails on real robots. As the authors note, the simulator itself is lying.
Compared with hand-crafted digital twins and 3D reconstruction approaches, action-conditioned video world models offer photorealism but unreliable reward signals — the exact gap World-VLA-Loop targets.
Core Architecture
- Closed-loop co-evolution: instead of training the world model once and freezing it, the VLA policy's failure rollouts are iteratively fed back to refine the world model, which in turn provides more accurate RL signals for the next policy iteration.
- State-aware video world model: the model predicts both future observations and a reward signal (success/failure), adding a discriminative reward prediction head on top of video generation.
- SANS dataset (Success And Near-Success trajectories): near-success trajectories (e.g., grasping an object but dropping it, placing it with slight offset) teach the model how tiny action changes flip outcomes — crucial for action-outcome alignment. Internet video is dominated by successes and rarely annotates failure causes, making this data scarce without a closed loop.
- Evaluated on LIBERO (Object, Goal, Long task suites) plus real-world robot tasks.
- Only two closed-loop iterations raise real-world success by 36.7% with minimal physical interaction — most learning happens in the virtual environment.
- Related work WoVR (Tsinghua / CASIA, arXiv:2602.13977) tackles the same frontier differently: KIR + masked GRPO + PACE with dual-channel action injection, focusing on a three-layer hallucination control stack, whereas World-VLA-Loop emphasizes the co-evolution loop and near-success data. Both confirm that hallucination control is the core bottleneck for video-world-model-based RL post-training.
- Long-horizon degradation: autocratic video models lose performance beyond ~200 frames (~20 seconds); better long-horizon backbones are needed.
- Sparse reward: currently final-state only; step-wise intermediate sub-goals are future work.
- Contact-rich tasks (insertion, screwing) where video diffusion poorly captures contact dynamics and errors compound.
- Out-of-distribution failure modes: the world model is only reliable on failures it has seen; new failure modes can re-trigger hallucination.
- Compute cost: video diffusion rollouts are far slower than 3D physics engines.
- "Minimal physical interaction" still requires manually collected initial data and real-robot validation each iteration — it reduces, not eliminates, real-world interaction.
- Liu, X., Bai, Z., Ci, H., Ma, K.Y., & Shou, M.Z. (2026). Closed-Loop Learning of Video World Model and VLA Policy. arXiv:2602.06508v1.
- Jiang, Z., Zhou, S., Jiang, Y., et al. (2026). WoVR: World Models as Reliable Simulators for Post-Training VLA Policies with RL. arXiv:2602.13977v1.
- Zhao, T., et al. (2024). LIBERO: Benchmarking Knowledge Transfer in Lifelong Robot Learning. arXiv:2311.03672.
Ablation Results (LIBERO-Object)
| Configuration | Task 1 | Task 2 | |---|---|---| | Visual alignment w/o near-success data | 60% | 65% | | Visual alignment (with near-success) | 85% | 95% | | Reward alignment w/o reward head | 60% | 70% | | Reward alignment (with reward head) | 75% | 90% |
Near-success data contributes +25–30 points to visual alignment; the reward head adds +15–20 points to reward alignment.
Training Loop
1. Step 0: train the initial world model on the SUPA dataset (collected success and near-success trajectories plus rollouts from an SFT OpenVLA-OFT baseline). 2. Step 1: RL post-train the policy in the world model, collect rollouts (including failures), augment SANS, retrain the world model. Baseline SFT policy: 13.3% → 36.7% (+23.4). 3. Step 2: repeat. Success rate reaches 50.0%.
Total: 13.3% → 50.0% (+36.7 points) in two iterations.
Evaluation
Analysis: Why SANS Matters
Success/failure are binary labels over a continuous action space; near-success trajectories sit in the gray zone and provide gradient information — which direction to nudge actions to succeed. SANS effectively acts as automatic curriculum generation: the policy's failure modes automatically produce near-success samples at exactly the difficulty frontier the model needs.
Limitations
Stated in the paper:
Deeper challenges:
Takeaway
World-VLA-Loop punctures the illusion that video world models are faithful simulators: they produce realistic video but hallucinate success under wrong actions, corrupting RL. The fix is a closed loop — failure rollouts iteratively refine the world model, and near-success data teaches the subtle boundary between almost-right and wrong. Two iterations lifted real-robot success from 13.3% to 50.0%, signaling a paradigm shift from "build a perfect simulator first" to agile, co-evolving simulators and policies.