Summary
This paper by Perry Dong, Ron Polonsky, Dorsa Sadigh, and Chelsea Finn (arXiv:2607.27203) examines whether Q-functions should be pretrained on offline data when fine-tuning a pretrained base policy with online reinforcement learning. Conventional wisdom suggests pretraining the Q-function is beneficial, but the authors show that naive Q-function pretraining often provides little benefit over random initialization. They attribute this to a fundamental mismatch: the pretrained Q-function targets the pretraining policy's value function, not the Q-function that online RL converges to, and this gap persists even after offline value maximization. Based on this finding, the paper proposes IPE (Integrated Policy Ensemble initialization), a simple method that trains multiple diverse policies and uses their aggregate rollouts to bootstrap Q-function learning during online RL. The work challenges a common assumption in value-based RL fine-tuning pipelines and offers a practical alternative for practitioners.
Paper Overview
Field: Machine Learning
Authors: Perry Dong, Ron Polonsky, Dorsa Sadigh, Chelsea Finn
Published: 2026-07-29
arXiv: 2607.27203
Summary
Pre-training followed by fine-tuning has become the dominant recipe for learning performant policies, and in value-based reinforcement learning (RL) this raises a natural question: given a pretrained policy, should the Q-function be pretrained on offline data too? Conventional wisdom suggests it should, but recent results show that online RL with a randomly-initialized Q-function can result in highly performant and reliable policies without needing to pretrain the Q-function.
Key Findings
- The authors systematically study whether pretraining the Q-function actually helps when fine-tuning on top of a pretrained base policy.
- Surprisingly, naive Q-function pretraining often provides little benefit over random initialization.
- This stems from a fundamental mismatch: the pretrained Q-function is learned for the pretraining policy, not for the Q-function that online RL fine-tuning converges to — and this gap persists even after offline value maximization.
Proposed Method: IPE
Based on this analysis, the paper introduces IPE (Integrated Policy Ensemble initialization), a simple approach:
- Train multiple diverse policies.
- Use their aggregate rollouts to bootstrap Q-function learning in online RL.
This provides a better starting point for online fine-tuning than naive Q-function pretraining, addressing the policy mismatch identified above.
---
*Auto-collected on 2026-07-31*
This page is an English static mirror generated for search and AI citation.
It may be a full translation or structured summary of the Chinese original.
Canonical interactive discussion lives on the Chinese page:
https://zhichai.net/topic/178503820