Q2RL: Extracting Q-functions from Behavior Cloning for On-Robot Reinforcement Learning

Forum topic · 小凯 · 2026-05-08

Summary

Q2RL is an offline-to-online reinforcement learning algorithm for robotics that builds on behavior cloning (BC). It addresses a common failure mode where existing offline-to-online methods replace previously learned good actions due to distribution mismatch between offline data and online learning. The method has two components: Q-Estimation, which extracts a Q-function from a BC policy using a few environment interaction steps before running online RL, and Q-Gating, which switches between BC and RL policy actions based on their respective Q-values to gather training samples. On manipulation tasks from the D4RL and robomimic benchmarks, Q2RL outperforms state-of-the-art offline-to-online baselines in success rate and convergence time. It is efficient enough for on-robot RL, learning robust policies for contact-rich, high-precision tasks such as pipe assembly and kitting within 1-2 hours of online interaction, achieving success rates up to 100% and up to 3.75x improvement over the original BC policy. Code and videos: https://pages.rai-inst.com/q2rl_website/. arXiv: https://arxiv.org/abs/2605.05172

Overview

Research area: Robotics Authors: Lakshita Dodeja, Ondrej Biza, Shivam Vats, Stephen Hart, Stefanie Tellex, Robin Walters, Karl Schmeckpeper, Thomas Weng Published: 2026-05-06 arXiv: 2605.05172

Abstract

Behavior Cloning (BC) has emerged as a highly effective paradigm for robot learning. However, BC lacks a self-guided mechanism for online improvement after demonstrations have been collected. Existing offline-to-online learning methods often cause policies to replace previously learned good actions due to a distribution mismatch between offline data and online learning.

In this work, the authors propose Q2RL (Q-Estimation and Q-Gating from BC for Reinforcement Learning), an algorithm for efficient offline-to-online learning. The method consists of two parts:

1. Q-Estimation — extracts a Q-function from a BC policy using a few interaction steps with the environment, followed by online RL. 2. Q-Gating — switches between BC and RL policy actions based on their respective Q-values to collect samples for RL policy training.

Results

Across manipulation tasks from the D4RL and robomimic benchmarks, Q2RL outperforms SOTA offline-to-online learning baselines on success rate and time to convergence.
Q2RL is efficient enough for on-robot RL, learning robust policies for contact-rich and high-precision manipulation tasks such as pipe assembly and kitting in 1–2 hours of online interaction.
Achieves success rates of up to 100% and up to 3.75x improvement over the original BC policy.