Why Curiosity Forgets: Persistent Worlds and Episodic Memory for 3D Exploration

Curiosity-driven reinforcement learning carries an appealing premise. Instead of waiting for sparse external rewards in long-horizon tasks, an agent generates its own intrinsic motivation from surprise, the gap between what its world model predicts and what actually happens. The idea traces back to Tolman's latent learning experiments and resurfaced in deep RL through the Intrinsic Curiosity Module (ICM) and related work. Yet anyone who has tried to deploy curiosity in photorealistic 3D environments has watched it fail in a particular and frustrating way: the agent gets stuck in loops, repeatedly visiting the same corner of a room while collecting fresh "novelty" rewards for places it has already seen.

The paper Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration by Goli, Kerr, Reda, Jacobson, Tagliasacchi, and Kanazawa offers a clean diagnosis of this failure and a concrete remedy. The argument is that curiosity collapses not because the principle is wrong, but because the underlying memory is amnesiac on two fronts at once.

The Two-Fold Amnesia Problem

The authors identify two distinct memory failures that compound each other.

The first is on the world model side. A method like ICM learns a predictive model that functions as a statistical prior over the agent's lifelong experience. It is not an episodic record of the specific scene. When the agent returns to a region it explored five minutes ago, the learned model has partially "forgotten" the fine details of that region because they have been averaged into a broad prior. The prediction error spikes again, and the agent receives a spurious novelty reward for a place it has genuinely already explored. Surprise stops being a reliable signal of true novelty.

The second is on the agent side. Without a memory of where it has recently been, the policy has no way to steer toward unexplored regions. It cannot distinguish "I came from this direction" from "I have never been here." The agent therefore oscillates, drawn back to whatever the world model currently flags as surprising.

The key insight is that these two problems reinforce each other. A non-persistent world model manufactures false novelty, and an amnesiac policy has no internal context to override that false signal. Fixing only one leaves the other to sabotage exploration.

The Architecture: 3DGS as Persistent Memory, Transformers as Episodic Context

The solution pairs a persistent world model with an episodic agent, and the engineering choices here are worth examining closely.

For the persistent world model, the authors use online 3D Gaussian Splatting (3DGS). As the agent moves, the reconstruction continuously accumulates and refines its representation of the scene. When the agent revisits a region, the 3DGS forward model can render that view from accumulated past experience, and the discrepancy between the rendered view and the actual observation defines the curiosity reward. Because the reconstruction persists, previously seen regions render accurately, the discrepancy is low, and the agent is not re-rewarded for old states. Surprise stays honest.

I find the framing of 3DGS as a "proxy" particularly important. The authors are explicit that the ideal world model would be a dynamic, action-conditioned video model refined through lifelong experience. They note that current models like LingBot-World fail to maintain scene consistency, generating an entirely different image after a simple in-place 360 degree turn. Spatial persistence is the bottleneck. 3DGS is chosen precisely because it solves persistence by construction; geometry and appearance are baked into an explicit representation that does not drift the way a neural video predictor does.

For the episodic agent, the policy is a transformer operating purely over a sequence of RGB observations. Each frame is encoded into a per-frame token by fusing patch embeddings with DINOv2 features via a learnable query. These tokens pass through causal temporal self-attention and a linear-attention module with a global hidden state before an actor-critic head emits the next action and a value estimate. The crucial design decision is that the agent forms its own internal episodic representation directly from pixels rather than from an explicit geometric map.

This is the part I think deserves the most attention. Prior active-mapping methods bypass the exploration problem by feeding the agent an explicit map, sometimes built from ground-truth depth. That guarantees spatial awareness but sacrifices end-to-end flexibility and semantic richness, and it ties the agent to geometrically definable tasks. The approach in this paper uses the persistent 3D model only during training to compute the curiosity reward; at deployment, the agent runs on RGB alone, with no depth sensor, no localization, and no explicit map. This asymmetry, heavy machinery in the loop during training and a lean policy at test time, is an elegant way to get the benefits of spatial persistence without paying its costs at inference.

The authors also report that scheduled, temporary injections of random behavior during RL training are sufficient to cross the reward-less stretches of long-horizon exploration. This produces an emergent behavior where the agent learns to traverse already-seen regions in order to reach novel branches of unexplored space, without imitation learning bootstrapping or hierarchical goal selection.

Results and What They Imply

Trained purely from curiosity on HM3D, the agent outperforms RL-based active-mapping baselines and generalizes zero-shot to Gibson and to out-of-distribution AI-generated 3D worlds never seen during training. Because the policy is end-to-end and map-free at test time, fine-tuning it for a few episodes on downstream rewards such as apple picking and image-goal navigation beats training from scratch.

The ablation that I consider most revealing is the one where the persistent memory is artificially capped to a short-term window. Exploration capability deteriorates significantly. This is the empirical confirmation of the central thesis: persistence is not a nice-to-have, it is the mechanism that keeps surprise honest. It also implicitly validates the choice of 3DGS over a windowed neural predictor.

My Take: Toward Uncertainty-Aware Curiosity

The current design defines curiosity as raw rendering discrepancy between the persistent reconstruction and the observation. That works well for chasing unseen geometry, but it treats all discrepancy as equally informative. There is a more principled target hiding here.

What I want to see next is a persistent world model that is uncertainty-aware, so that intrinsic reward tracks the epistemic uncertainty of the reconstruction rather than its raw prediction error. In a 3DGS reconstruction, some regions are well-constrained by many consistent observations, while others are reconstructed from a single oblique glance and remain ambiguous. A reward keyed to epistemic uncertainty would let the agent revisit deliberately when its map is likely wrong, instead of only seeking geometry it has never seen. Curiosity would then become a form of active error correction, closing the loop between exploration and reconstruction quality.

This connects to a broader tension in intrinsic motivation research: aleatoric versus epistemic uncertainty. Raw prediction error conflates the two, which is exactly how the classic "noisy TV" problem arises, an agent endlessly rewarded by stochastic but uninformative observations. An uncertainty-aware persistent world model offers a natural disentanglement, because the reconstruction itself can represent how confident it is about each part of the scene.

Limitations and Open Questions

The reliance on 3DGS assumes static scenes. The persistence that makes the method work is precisely what breaks under dynamics, since a moving object would register as permanent surprise. Extending this to dynamic worlds will likely require the action-conditioned video models the authors themselves point to as the eventual goal, which reintroduces the persistence problem they currently sidestep.

There are also open questions about scaling. Online 3DGS during training is computationally heavy, and how this cost grows with scene size and episode length is not trivial. Finally, the transformer policy's episodic context is bounded by its attention window and hidden state; for very long horizons across large environments, whether the learned internal map degrades gracefully remains to be seen.

Still, the core contribution is durable. By cleanly separating persistent spatial memory from episodic agent context, this work shows that curiosity does not fail because the principle is flawed. It fails when memory fades. Keep the world model honest, give the agent a sense of where it has been, and curiosity scales to the kinds of photorealistic 3D environments where we actually want it to work.