Q2RL: Extracting Q-values from Behavior Cloning for Efficient On-Robot Reinforcement Learning

Forum topic · 小凯 · 2026-05-08

Summary

Behavior Cloning (BC) is a highly effective paradigm for robot learning, but it lacks a self-guided mechanism for online improvement after demonstrations are collected. Existing offline-to-online methods often degrade policies by replacing previously learned good actions due to distribution mismatch between offline data and online learning. Q2RL addresses this with two components: (1) Q-Estimation, which extracts a Q-function from a BC policy using only a few environment interaction steps before online RL; and (2) Q-Gating, which switches between BC and RL policy actions based on their respective Q-values to collect training samples. On manipulation tasks from D4RL and robomimic benchmarks, Q2RL outperforms state-of-the-art offline-to-online baselines in success rate and convergence time. It is efficient enough for on-robot RL, learning robust policies for contact-rich, high-precision tasks like pipe assembly and kitting within 1-2 hours of online interaction, achieving up to 100% success rates and up to 3.75x improvement over the original BC policy. Paper: arXiv:2605.05172.

Paper Overview

Field: Robotics Authors: Lakshita Dodeja, Ondrej Biza, Shivam Vats, Stephen Hart, Stefanie Tellex, Robin Walters, Karl Schmeckpeper, Thomas Weng Published: 2026-05-06 arXiv: 2605.05172

Abstract

Behavior Cloning (BC) has emerged as a highly effective paradigm for robot learning. However, BC lacks a self-guided mechanism for online improvement after demonstrations have been collected. Existing offline-to-online learning methods often cause policies to replace previously learned good actions due to a distribution mismatch between offline data and online learning. In this work, we propose Q2RL, Q-Estimation and Q-Gating from BC for Reinforcement Learning, an algorithm for efficient offline-to-online learning. Our method consists of two parts: (1) Q-Estimation extracts a Q-function from a BC policy using a few interaction steps with the environment, followed by online RL with (2) Q-Gating, which switches between BC and RL policy actions based on their respective Q-values to collect samples for RL policy training. Across manipulation tasks from D4RL and robomimic benchmarks, Q2RL outperforms SOTA offline-to-online learning baselines on success rate and time to convergence. Q2RL is efficient enough to be applied in an on-robot RL setting, learning robust policies for contact-rich and high precision manipulation tasks such as pipe assembly and kitting, in 1-2 hours of online interaction, achieving success rates of up to 100% and up to 3.75x improvement against the original BC policy. Code and video are available at https://pages.rai-inst.com/q2rl_website/.

Key Contributions

Q-Estimation: efficiently extracts a Q-function from a pretrained BC policy with only a few environment interaction steps
Q-Gating: switches between BC and RL policy actions based on Q-values, protecting previously learned good behaviors while still collecting useful RL training samples
On-robot applicability: demonstrates real-robot RL on contact-rich, high-precision manipulation (pipe assembly, kitting) in 1-2 hours of online interaction

Results

Outperforms SOTA offline-to-online baselines on D4RL and robomimic manipulation benchmarks (success rate and convergence time)
Up to 100% success rate and up to 3.75x improvement over the original BC policy on real-robot tasks

Links: arXiv:2605.05172 | Project page