Beyond Imitation: RL-Based Sim-Real Co-Training for Vision-Language-Action Models

Forum topic · 小凯 · 2026-04-01

Summary

This forum post analyzes the paper 'Beyond Imitation: Reinforcement Learning-Based Sim-Real Co-Training for Vision-Language-Action Models' (arXiv:2602.12628), authored by Jiafei Duan et al. from Tsinghua University, Shanghai Jiao Tong University, and other institutions. Vision-Language-Action (VLA) models like OpenVLA traditionally rely on imitation learning, which suffers from expensive data collection, distribution shift, and a performance ceiling. On unseen tasks such as button pressing and knob turning, OpenVLA succeeds less than 40% of the time. The paper proposes Sim-Real Co-Training, in which separate simulation and real-world policies are trained in parallel in environments like Genesis and Isaac Gym, exchanging knowledge through a bidirectional distillation mechanism called Dream-Grafting. An automatic reward model built on a VLM plus success detectors replaces hand-crafted rewards. Reported results show average success rate rising from 44.4% to 75.0% (+30.6%), with the out-of-distribution generalization gap shrinking from -41.7% to -12.2%. Ablations identify bidirectional distillation, separated policy networks, and the automatic reward model as key contributors. Limitations include dependence on simulation fidelity, VLM reward precision for fine manipulation, and high GPU cost (~72 hours on 8x A100 plus 4 Franka robots).

Beyond Imitation: RL-Based Sim-Real Co-Training for VLA Models — Research Notes

This post is a detailed Chinese-language research digest of the paper "Beyond Imitation: Reinforcement Learning–Based Sim–Real Co-Training for Vision-Language-Action Models" (arXiv:2602.12628, released February 18, 2026), by Jiafei Duan, Yuhan Liu, Haoyi Niu, et al., from Tsinghua University, Shanghai Jiao Tong University, Fudan University, UCSB, Shanghai AI Lab, and BAAI. Below is a structured English summary of the key findings.

Key points

Problem: the VLA data famine

VLA models (e.g., OpenVLA, π0) map vision + language instructions to robot actions, typically trained via imitation learning (behavior cloning).
Bottlenecks: costly human demonstration data, distribution shift on unseen scenarios, and a ceiling — imitation cannot exceed human performance.
OpenVLA reaches 80%+ success on common tasks but under 40% on unseen manipulations (button pressing, knob turning).
Simulation offers unlimited data, but the sim-to-real gap makes naive transfer or naive data mixing degrade performance.

Method: Sim-Real Co-Training with Dream-Grafting

Two separate policy networks (sim and real) are optimized in parallel; simulation uses Genesis / Isaac Gym with ~1024 parallel environments, real training uses 4 Franka Panda robots.
Dream-Grafting (bidirectional distillation): the sim policy learns physical intuition from the real policy; the real policy learns manipulation skills from the sim policy. Policies are updated asynchronously to avoid gradient conflicts.
Automatic reward model: a VLM-based state describer + rule-based success detector + process heuristics replace hand-designed reward functions, generalizing zero-shot to new tasks.
Architecture: frozen DINOv2 vision encoder, frozen LLaMA-3.1 language encoder, Perceiver Resampler fusion, and a Diffusion Policy (DiT) action head (chosen for multimodal action distributions).
Training: AdamW, lr 1e-4, batch 256 (sim:real = 3:1), distillation temperature τ = 0.5, distillation weight λ = 0.3; total ~72 hours on 8× A100 — roughly 10×+ more efficient than weeks of real-world demonstration collection.

Results

OpenVLA improvements with co-training:

| Task type | Baseline | Co-Training | Gain | |---|---|---|---| | Pick and place | 75.3% | 91.2% | +15.9% | | Drawer operation | 42.1% | 78.6% | +36.5% | | Button pressing | 31.5% | 67.8% | +36.3% | | Knob turning | 28.7% | 62.4% | +33.7% | | Average | 44.4% | 75.0% | +30.6% |

Out-of-distribution generalization:

Pure imitation: seen 80.2% / unseen 38.5% (gap −41.7%)
Sim pretrain + real fine-tune: seen 77.8% / unseen 51.2% (gap −26.6%)
Sim-Real Co-Training: seen 81.5% / unseen 69.3% (gap −12.2%)

Ablations: removing bidirectional distillation costs −13.8%; using manual rewards instead of the VLM reward model costs −16.3%; sharing one policy network costs −19.6%; pure sim or pure real training lose −32.9% / −30.6%.

Qualitative insights

Knob turning: sim exploration over varied friction yields robust rotation strategies transferred to reality.
Drawer operation: real-world feedback about excessive force corrects the sim reward model (real sims lack damping).

Limitations and future directions

Degrades when sim-to-real gaps are large (cloth, fluids); domain randomization still needed as a complement.
VLM rewards may lack precision for fine-grained manipulation (e.g., peg-hole alignment).
High compute cost limits reproducibility for small labs.
Future work: adaptive simulation parameter tuning, hierarchical co-training (low-level skills in sim, high-level planning in real), and cross-robot transfer.

Core insight

References

Duan et al. (2026). Beyond Imitation: RL-Based Sim-Real Co-Training for VLA Models. arXiv:2602.12628.
Kim et al. (2024). OpenVLA: An Open-Source Vision-Language-Action Model.
Genesis Team (2024). A Generative Physics Engine for Robotics and Beyond.
Chi et al. (2023). Diffusion Policy: Visuomotor Policy Learning via Action Diffusion.