Paper Overview
Field: CV Authors: Likui Zhang, Zhe Zhu, Zhihao Zhan, Yibo Bai, Hanlin Wang, Dongwei Li, Bin Wang, Tianle Zhang, Xiaodan Liang, Liang Lin Published: 2026-10-21 arXiv: 2610.30097
Summary
Vision-language-action (VLA) models have shown promise in robotic manipulation, but imitation learning (IL) alone often struggles to master complex long-horizon skills. Reinforcement learning (RL) offers an effective avenue for further improving VLA policies. However, existing VLA-RL methods typically suffer from delayed and biased credit assignment due to trajectory-level sparse rewards, while imbalanced subtask mastery creates bottlenecks that limit overall performance.
To address these challenges, the authors propose AtomicRL, an RL post-training framework that leverages atomic task structures for reliable credit assignment and adaptive exploration.
Key Components
- Semantic-Aware Policy Optimization (SAPO): uses semantic-aware gating to constrain TD residuals from failed atomic subtasks from propagating across task boundaries, mitigating credit pollution while preserving within-subtask credit assignment and value guidance.
- Bottleneck-Adaptive Task Exploration (BATE): dynamically coordinates task sampling and local exploration incentives based on each atomic subtask's mastery level, directing learning toward remaining bottlenecks.
- Real2Sim2Real pipeline: connects online RL in simulation with real-world deployment, facilitating practical transfer to physical robots.
- LIBERO-LONG: AtomicRL achieves absolute performance gains of 3.6% and 4.0% over PPO on two VLA architectures.
- CALVIN ABC-D: full-task performance improves by up to 9.5% over PPO.
- Real-world transfer: combined with the Real2Sim2Real pipeline, AtomicRL further improves real-world task performance by 21.3% on average.
Results
--- *Auto-collected on 2026-10-23*