Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning? (IPE, arXiv 2607.27203)

Forum topic · 小凯 · 2026-07-31

Summary

This paper by Perry Dong, Ron Polonsky, Dorsa Sadigh, and Chelsea Finn (Stanford) systematically examines whether Q-function pretraining helps when fine-tuning value-based RL policies online. Contrary to conventional wisdom, the authors find that naively pretraining the Q-function on offline data often provides little benefit over random initialization, even though pretraining plus fine-tuning is the dominant recipe for learning performant policies. They trace this to a fundamental mismatch: the Q-function learned during offline pretraining targets the pretrained policy's value estimates, not the Q-function that online RL fine-tuning converges to, and this gap persists even after offline value maximization. Based on this insight, they propose IPE (Integrated Policy Ensemble initialization), a simple method that trains several diverse policies and uses their aggregated rollouts to bootstrap Q-function learning during online RL. The work was published on arXiv as 2607.27203 and discussed on zhichai.net.

Paper Overview

Field: Machine Learning Authors: Perry Dong, Ron Polonsky, Dorsa Sadigh, Chelsea Finn Published: 2026-07-29 arXiv: 2607.27203

Summary

Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) this raises a natural question: given a pretrained policy, should the Q-function be pretrained on offline data too? Conventional wisdom suggests it should, but recent results show that online RL with a randomly-initialized Q-function can result in highly performant and reliable policies without needing to pretrain the Q-function.

In this paper, the authors systematically study whether pretraining the Q-function actually helps when fine-tuning on top of a pretrained base policy. Surprisingly, they find that naive Q-function pretraining often provides little benefit over random initialization.

They show this stems from a fundamental mismatch: the Q-function learned during pretraining targets the pretrained policy's value estimates, not the Q-function that online RL fine-tuning converges to, and this gap persists even after offline value maximization.

Proposed Method: IPE

Based on this analysis, the authors propose IPE (Integrated Policy Ensemble initialization), a simple approach:

Train multiple diverse policies during the pretraining stage
Use their aggregated (pooled) rollouts to bootstrap Q-function learning during online RL

Full Abstract

> Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) this raises a natural question: given a pretrained policy, should the Q-function be pretrained on offline data too? Conventional wisdom suggests it should, but recent results show that online RL with a randomly-initialized Q-function can result in highly performant and reliable policies without needing to pretrain the Q-function. In this paper, we systematically study whether pretraining the Q-function actually helps when fine-tuning on top of a pretrained base policy. We find, surprisingly, that naive Q-function pretraining often provides little benefit over random initialization. We show this stems from a fundamental mismatch: the Q-function...

---

*Auto-collected on 2026-07-31.*

Links: arXiv:2607.27203

Do You Really Need to Pretrain Q-Functions for Online RL Fine-Tuning? (IPE, arXiv 2607.27203)

Summary

Paper Overview

Summary

Proposed Method: IPE

Full Abstract

Tags