Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration

Forum topic · 小凯 · 2026-05-25

Summary

This paper investigates why curiosity-driven reinforcement learning often fails in complex photorealistic 3D environments, attributing the failure to a lack of spatial persistence and episodic context. The authors show that effective intrinsic motivation requires a persistent, continuously updated world model combined with an agent that maintains an episodic history to navigate toward novel regions. They instantiate this with online 3D reconstruction as the persistent world model and a sequential model over RGB observations as the policy. Trained purely with curiosity on HM3D, the agent outperforms RL-based active mapping baselines and zero-shot generalizes to Gibson and AI-generated worlds. The end-to-end policy adapts efficiently to downstream tasks such as apple picking and image-goal navigation, beating training-from-scratch baselines, while requiring only RGB frames at deployment.

Paper Overview

Field: Machine Learning Authors: Lily Goli, Justin Kerr, Daniele Reda Published: 2026-05-25 arXiv: 2505.14488

Abstract (Translation)

Exploration is a prerequisite for learning useful behaviors in sparse-reward, long-horizon tasks, especially in 3D environments. Curiosity-driven reinforcement learning addresses this via intrinsic rewards arising from mismatches between an agent's predictive model of the world and reality. However, translating such intrinsic motivation to complex, photorealistic environments remains difficult: agents can get stuck in local loops and repeatedly receive novelty rewards for revisiting forgotten states.

In this work, the authors demonstrate that this failure stems from a lack of spatial persistence and episodic context. They show that effective curiosity requires a persistent, continuously updated world model, coupled with an agent that maintains an episodic trajectory history in order to navigate toward novel regions.

They use online 3D reconstruction as the persistent world model, while the agent's policy is parameterized as a sequential model over RGB observations to retain episodic context. This design enables effective exploration during training while allowing the agent to navigate at deployment time using only RGB frames.

Trained purely with curiosity on HM3D, the agent outperforms RL-based active mapping baselines and zero-shot generalizes to Gibson and AI-generated worlds. The end-to-end policy adapts efficiently to downstream tasks such as apple picking and image-goal navigation, outperforming training-from-scratch baselines.

Key Contributions

Identifies missing spatial persistence and episodic context as the cause of curiosity-driven exploration failures in photorealistic 3D environments
Uses online 3D reconstruction as a persistent world model
Employs a sequential-model policy over RGB observations for episodic context
RGB-only deployment; zero-shot transfer to Gibson and AI-generated worlds
Efficient adaptation to downstream tasks (apple picking, image-goal navigation)

*Auto-collected on 2026-05-25*