Paper Overview
Field: Machine Learning Authors: Mohamed Nabail, Leo Cheng, Jingmin Wang Published: 2026-06-19 arXiv: 2506.14974
Summary
Preference-based RL provides an approach to learning reward models from pairwise comparisons of behaviors, bypassing the need for explicit reward design. However, existing methods typically rely on passive data collection and suffer from poor sample efficiency, especially during the early stages of learning.
The authors introduce a model-based approach that actively directs exploration by jointly reasoning over uncertainties in the reward, dynamics, and value functions. Their method, Uncertainty-Balanced Preference Planning (UBP2), uses ensembles of reward, dynamics, and value function models to evaluate candidate trajectories according to a unified score that combines expected reward, terminal value, and epistemic uncertainty. Planning under this objective yields an explicit tradeoff between exploitation and information acquisition without requiring ad hoc exploration heuristics.
Key Contributions
- Active exploration: Joint uncertainty reasoning over reward, dynamics, and value functions replaces passive data collection.
- Unified planning objective: A single score combining expected reward, terminal value, and epistemic uncertainty from model ensembles.
- Theoretical guarantees: Sublinear regret guarantees for both finite-horizon and infinite-horizon settings under standard regularity assumptions.
- Empirical results: On the Meta-World benchmark, UBP2 achieves substantially higher sample efficiency than model-free preference-based methods and non-optimistic model-based baselines.