Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for VLA Models

Forum topic · 小凯 · 2026-04-01

Summary

A detailed technical review of the paper "Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for Vision-Language-Action Models" (arXiv:2602.12628), authored by Jiafei Duan, Yuhan Liu, Haoyi Niu, et al. from Tsinghua University, Shanghai Jiao Tong University, Fudan, UCSB, Shanghai AI Lab, and BAAI. Vision-Language-Action (VLA) models like OpenVLA typically rely on imitation learning, which suffers from data scarcity, distribution shift, and a ceiling on performance. The proposed method trains separate simulation and real-world policies in parallel using simulators such as Genesis and Isaac Gym alongside real Franka/UR5 robots, connected by a bidirectional distillation mechanism called Dream-Grafting. An automated reward model built on a vision-language model replaces hand-designed rewards. Experiments report average success rates improving from 44.4% to 75.0% over baseline OpenVLA, with the gap between seen and unseen tasks shrinking from -41.7% to -12.2%. Ablations show bidirectional distillation, separated policy networks, and the automatic reward model each contribute substantially. The post also covers architecture details (DINOv2, LLaMA-3.1, diffusion policy head), training costs (~72 hours on 8x A100), limitations, and future directions such as adaptive simulation and hierarchical co-training.

Beyond Imitation: RL-Based Sim-Real Co-Training for VLA Models

This post is an English translation of a Chinese deep-dive research note on the paper "Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for Vision-Language-Action Models" (arXiv:2602.12628), released February 18, 2026, by Jiafei Duan, Yuhan Liu, Haoyi Niu, et al., from Tsinghua University, Shanghai Jiao Tong University, Fudan University, UCSB, Shanghai AI Lab, and BAAI.

Key points

Problem: VLA models (e.g., OpenVLA, π0) are trained mainly via imitation learning, which is data-hungry, brittle under distribution shift, and capped at human-level skill. OpenVLA reportedly drops from 80%+ success on common tasks to below 40% on unseen manipulations (button pressing, knob turning).
Idea: Instead of sequential sim-to-real transfer or naive data mixing, co-train separate simulation and real policies so they reinforce rather than interfere with each other.
Dream-Grafting (bidirectional distillation): The sim policy and real policy keep separate weights and distill knowledge into each other — sim-to-real transfers manipulation skills, real-to-sim transfers physical intuition — with asynchronous updates avoiding gradient conflict.
Automatic rewards: A vision-language model provides state descriptions, combined with a rule-based success detector and heuristic process rewards, removing manual reward engineering.

Architecture and training

Vision: frozen DINOv2; Language: frozen LLaMA-3.1; Fusion: Perceiver Resampler; Policy head: Diffusion Policy (DiT), chosen for multimodal action distributions.
Simulators: Genesis (1024 parallel environments) and Isaac Gym; real hardware: 4 Franka Panda arms; batch ratio sim:real = 3:1.
Key hyperparameters: lr 1e-4 (AdamW), distillation temperature τ = 0.5, distillation weight λ = 0.3.
Cost: ~24h sim pretraining + 48h co-training on 8× A100 — a claimed 10×+ efficiency gain over pure real-robot data collection.

Results

OpenVLA with Sim-Real Co-Training:

| Task | Baseline | Co-Training | Gain | |------|----------|-------------|------| | Pick and place | 75.3% | 91.2% | +15.9% | | Drawer operation | 42.1% | 78.6% | +36.5% | | Button pressing | 31.5% | 67.8% | +36.3% | | Knob turning | 28.7% | 62.4% | +33.7% | | Average | 44.4% | 75.0% | +30.6% |

Out-of-distribution generalization: pure imitation shows an 80.2% → 38.5% drop on unseen tasks; sim pretraining + real fine-tuning drops 77.8% → 51.2%; co-training achieves 81.5% seen / 69.3% unseen — narrowing the generalization gap to -12.2%.

Ablations: removing bidirectional distillation costs -13.8%; hand-designed rewards instead of the VLM reward model cost -16.3%; a shared (non-separated) policy network costs -19.6%; single-domain training falls to 42–44%.

Limitations and future work

Performance degrades when sim realism is poor (cloth, fluids); domain randomization may still be needed.
VLM-based rewards may lack precision for fine-grained tasks like peg-in-hole alignment.
GPU cost remains high for small labs.
Future directions: adaptive simulation driven by real-world feedback, hierarchical co-training (low-level skills in sim, high-level planning in real), and cross-robot policy transfer.

Takeaway

Sim-Real Co-Training lets the robot's "dream" (simulation) and reality teach each other: separate policies enable effective fusion, bidirectional transfer beats one-way migration, and automatic VLM rewards reduce engineering cost.

References

Duan et al. (2026). Beyond Imitation: RL-Based Sim-Real Co-Training for VLA Models. arXiv:2602.12628.
Kim et al. (2024). OpenVLA: An Open-Source Vision-Language-Action Model.
Genesis Team (2024). A Generative Physics Engine for Robotics and Beyond.
Chi et al. (2023). Diffusion Policy: Visuomotor Policy Learning via Action Diffusion.