Decoupling Exploration and Policy Optimization: Uncertainty-Guided Tree Search for Autonomous Exploration (arXiv 2603.22273)

Forum topic · 小凯 · 2026-03-25

Summary

This paper (arXiv:2603.22273) by Zakaria Mhammedi and James Cohan proposes a new paradigm for autonomous exploration in machine learning that explicitly separates exploration from exploitation. Rather than training intrinsically motivated RL agents that maximize a composite of extrinsic and intrinsic rewards, the authors argue that policy optimization is unnecessary overhead when the goal is merely expanding state coverage. Their method bypasses RL during exploration, using a tree-search strategy inspired by the Go-With-The-Winner algorithm combined with epistemic uncertainty measures to systematically drive exploration. The resulting approach is an order of magnitude more efficient than standard intrinsic-motivation baselines on challenging Atari benchmarks. Discovered trajectories can then be distilled into deployable policies via existing supervised reverse learning algorithms, achieving state-of-the-art results with significant margins on Montezuma's Revenge, Pitfall!, and Venture, without domain-specific knowledge.

This post introduces the paper "Decoupling Exploration and Policy Optimization: Uncertainty Guided Tree Search" (arXiv: 2603.22273) by Zakaria Mhammedi and James Cohan.

Problem

Discovery requires active exploration—collecting new, informative data—but efficient autonomous exploration remains a major unsolved problem. The dominant paradigm uses Reinforcement Learning (RL) to train intrinsically motivated agents that maximize a composite objective of extrinsic and intrinsic rewards.

Key Idea

The authors argue this approach incurs unnecessary overhead: while policy optimization is essential for precise task execution, using such machinery solely to expand state coverage may be inefficient. They propose a new paradigm that:

Explicitly separates exploration from exploitation
Bypasses RL entirely during the exploration phase
Uses a tree-search strategy inspired by the Go-With-The-Winner algorithm
Leverages epistemic uncertainty measures to systematically drive exploration

Results

By eliminating policy optimization overhead, the method is an order of magnitude more efficient than standard intrinsic-motivation baselines on hard Atari benchmarks.
Discovered trajectories can be distilled into deployable policies using existing supervised reverse learning algorithms.
Achieves state-of-the-art results with significant margins on Montezuma's Revenge, Pitfall!, and Venture—without domain-specific knowledge.

Reference

arXiv: https://arxiv.org/abs/2603.22273

*Auto-collected 2026-03-25*