*An editorial walkthrough (originally in Chinese) of the paper "Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration" by Lily Goli, Justin Kerr, and Daniele Reda. Project page: https://recuriosity.github.io/*
Key points
Why curiosity-driven RL gets stuck circling
- In reinforcement learning, "curiosity" is formalized as intrinsic reward: the agent is rewarded when its prediction error is high, i.e., when the world surprises it.
- In photorealistic 3D indoor environments (corridors, rooms, furniture, changing light), this mechanism breaks down: agents fall into local loops, repeatedly traversing the same hallway or pacing the same room.
- Cause: the agent's feature representations are overly sensitive to viewpoint and lighting. The same place seen in the morning and at dusk looks "new," so the agent rewards itself for rediscovery. Some designs even reward revisited states—rewarding the agent's own forgetting.
- The paper identifies two root deficits:
- Spatial persistence — a stable, objective, external model of the world that doesn't reset when the agent looks away or returns a day later.
- Episodic context — a temporal, first-person record of *how* the agent moved through space, enabling decisions like "I've been here three times; try the unexplored path."
- Trained purely with curiosity (no external task rewards) on HM3D (Habitat-Matterport 3D, photorealistic scanned indoor scenes), the agent outperforms all RL-based active mapping baselines.
- Zero-shot generalization: strong performance on unseen Gibson environments and on procedurally generated (AI-generated) worlds—the agent learned a transferable exploration methodology, not memorized landmarks.
- Downstream transfer:
- *Apple picking*: initializing from the curiosity-pretrained policy then fine-tuning significantly beats training from scratch.
- *Image-goal navigation*: navigating to a target shown as an image also improves markedly, with better spatial reasoning and backtracking.
- Key insight: in sparse-reward, long-horizon environments, purposeless exploration is not wasted—it accumulates spatial priors (the 3D map and trajectory memory) that become the knowledge foundation for all later tasks.
- Title: *Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration*
- Authors: Lily Goli, Justin Kerr, Daniele Reda
- arXiv category: cs.LG
- Project page / demos: https://recuriosity.github.io/
The fix: two mutually reinforcing pillars
1. Online 3D reconstruction as a persistent world model. As it explores, the agent incrementally builds a real, geometric 3D scene model—walls are walls, doors are doors. "Have I been here?" is answered deterministically by querying geometry ("is this location in the visited set?"), not by fragile neural-feature similarity. This eliminates the classic *revisiting-forgotten-states* failure. 2. A sequence model as the policy. Instead of a memoryless feedforward policy acting on each RGB frame, the policy is a Transformer-like sequence model over consecutive RGB observations, giving the agent *trajectory awareness*—where it came from and what path it has taken.
Together: spatial persistence answers *"what does the world look like?"*; episodic context answers *"what path have I taken through it?"* Either alone is insufficient.
Results
The paradox: you must remember to be curious
The title's double meaning—both "remember to stay curious" and "memory is what enables curiosity"—captures the paper's core claim. Without memory, curiosity is blind wandering; with memory, it becomes directed, efficient, and productive. The agent's memory is deliberately structured and selective: the 3D reconstruction stores geometry (not pixels), the sequence model stores narrative context (not every frame)—memory in service of curiosity, not a burden on it.
Paper info: