Paper Overview
Field: ML Authors: Lily Goli, Justin Kerr, Daniele Reda Published: 2026-05-25 arXiv: 2505.14488
Summary
Exploration is a prerequisite for learning useful behavior in sparse-reward, long-horizon tasks, especially in 3D environments. Curiosity-driven reinforcement learning addresses this via intrinsic rewards generated from mismatches between an agent's predictive model of the world and reality. However, translating this intrinsic motivation to complex, photorealistic environments remains difficult because agents may fall into local loops and repeatedly gain fresh rewards from forgotten, re-visited states.
In this work, the authors demonstrate that this failure stems from a lack of spatial persistence and episodic context. They show that effective curiosity requires:
- A persistent, continuously updated world model
- An agent that maintains episodic trajectory history to navigate toward novel regions
- Trained purely via curiosity on HM3D, the agent outperforms RL-based active mapping baselines
- Zero-shot generalization to Gibson and AI-generated worlds
- The end-to-end policy adapts efficiently to downstream tasks such as apple picking and image-goal navigation, outperforming from-scratch training baselines
Their design uses online 3D reconstruction as the persistent world model, while the agent policy is parameterized as a sequence model over RGB observations to retain episodic context. This enables effective exploration during training while allowing the agent to navigate at deployment using only RGB frames.