Why Experience Replay Breaks in Continual Reinforcement Learning

Introduction

There is a comfortable assumption in machine learning that techniques which work in one setting will generalize to adjacent settings. Replay buffers are a textbook example. In supervised continual learning, the recipe for fighting catastrophic forgetting is well established: store some fraction of past data, interleave it with current data during training, and the network retains its old skills. The intuition is simple and the empirical support is strong. So when researchers face the same forgetting problem in continual reinforcement learning, the natural first move is to reach for replay.

The paper Replay-enhanced Continual Reinforcement Learning, which introduces a method called RECALL, makes a quietly important point: this transfer does not work cleanly, and it fails even under the most generous conditions imaginable. The authors consider the case of perfect memory, where every transition from every previous task is available during the current task. In supervised learning, perfect memory essentially solves forgetting by construction; you are simply doing joint training. In RL, perfect memory still fails. Understanding why reveals something general about the difference between supervised objectives and the optimization dynamics of reinforcement learning.

The Two Failure Modes

The paper identifies two distinct mechanisms behind the failure, and they are worth separating because they have different cures.

The reward scale problem. Most RL algorithms are not invariant to the scale of rewards. Value functions estimate expected discounted return, and the magnitude of the targets used in temporal difference updates depends directly on the reward scale of the task. Now imagine a continual setting where the agent has already mastered several tasks. Those tasks produce high returns because the policy is good at them. The new task, by contrast, starts near zero return because the agent has not learned it yet. When you mix old and new data in a single replay-augmented update, the gradients associated with the well-learned, high-reward tasks dominate. The optimization process treats those tasks as more salient simply because their numbers are bigger.

The consequence is a loss of plasticity. The agent keeps polishing what it already knows and largely ignores the new task, because the learning signal from the new task is drowned out. This is the opposite of the usual forgetting story. The network is not forgetting old tasks; it is failing to learn the new one. That distinction matters because it is invisible if you only measure average performance, and it would never show up in a supervised replay setting where all classes contribute comparably scaled loss terms.

The distributional shift problem. The second failure is closer to a classic offline RL pathology. When you replay transitions from an old task while the current policy has drifted, you are effectively doing offline learning on a dataset collected by a stale policy. The mismatch between the behavior distribution in the buffer and the current learned policy induces extrapolation errors in value estimation. This is the same difficulty that motivates the entire offline RL literature, where naively bootstrapping from out-of-distribution actions leads to overestimation and collapse. In the continual setting, this distributional shift manifests as forgetting of the old tasks, even though their data is sitting right there in the buffer.

So replay in RL is squeezed from both sides. Reward scale costs you plasticity on the new task, and distributional shift costs you stability on the old tasks. Either one is enough to undermine the method.

How RECALL Addresses Both

RECALL is deliberately a two-part fix, with each component targeting one failure mode.

• Adaptive normalization on approximate targets equalizes the saliency of tasks. By normalizing the value targets, the method removes the dependence on absolute reward magnitude, so a new task with small returns and an old task with large returns contribute learning signal on comparable footing. This is conceptually related to PopArt, the technique introduced for learning values across many scales of magnitude, and it is a sensible reuse of that idea in the continual context. The effect is to restore plasticity on the new task.

• Policy distillation on old tasks addresses stability. Rather than relying on raw value bootstrapping over replayed transitions, the agent distills the previously learned policy behavior, which sidesteps the extrapolation error that arises from the policy and data distribution mismatch. Distillation gives a more stable supervisory signal for the old tasks because it regresses toward known good action distributions rather than chasing potentially overestimated bootstrap targets.

The authors evaluate on the Continual World benchmark, a standard suite built on Meta-World manipulation tasks that has become the common testbed for continual RL. The headline empirical result is that RECALL substantially outperforms pure perfect memory replay, and matches or exceeds state of the art continual learning methods on overall performance. The comparison against perfect memory is the most instructive number, because it isolates the contribution of the two corrections rather than the contribution of having more data.

My Take

The detail I find most revealing is that the baseline here is perfect memory, not a small or imperfect buffer. In most replay research, the interesting questions are about memory efficiency: how little can you store and still avoid forgetting. The implicit assumption is always that more memory monotonically helps and that infinite memory is the ceiling. This paper quietly demolishes that assumption for RL. Even with everything stored, plain replay underperforms, which means the problem was never fundamentally about memory at all. It was about the objective.

This reframes the whole conversation. Catastrophic forgetting in RL is partly a symptom of unnormalized, non-stationary optimization targets rather than a pure data retention problem. That has a useful corollary: any method that hopes to import a supervised continual learning trick into RL should first ask whether the trick implicitly assumed comparably scaled, stationary, and on-distribution training signals. Many of them do.

I would push the analysis one step further than the paper. The reward scale issue is really an instance of a broader pathology in multi-objective and multi-task optimization where gradient magnitudes encode unintended priorities. We see versions of this in multi-task learning with gradient surgery methods, in RLHF where reward model scale interacts badly with KL penalties, and in any setting where heterogeneous loss terms are summed without normalization. The adaptive normalization in RECALL is a specific cure, but the underlying lesson is that RL practitioners should treat the relative scale of learning signals as a first-class design variable, not an implementation detail.

There are limitations worth stating honestly. The perfect memory assumption is a clean analytical device, but it is unrealistic at scale, and it remains an open question how the two RECALL mechanisms degrade as the buffer shrinks. Policy distillation also presupposes that you can identify and store a faithful representation of the old policy, which has its own storage and architecture costs. Continual World, while standard, is a manipulation benchmark with a particular structure of task similarity; whether the same fixes hold under sharper distribution shifts, for instance across genuinely different control morphologies, is not established here.

Conclusion

The lasting contribution of Replay-enhanced Continual Reinforcement Learning is diagnostic as much as it is methodological. It shows that the comforting transfer of replay from supervised learning to RL does not survive contact with the optimization dynamics that make RL distinctive: reward scale sensitivity and distributional shift. RECALL fixes both with adaptive target normalization and policy distillation, and the empirical results on Continual World support the prescription.

The open question I would most like to see answered is whether these two corrections are sufficient or merely necessary. Reward scale and distributional shift are the two failure modes this paper found, but the space of ways RL objectives differ from supervised ones is larger. As continual and lifelong RL move toward agents that accumulate skills over long horizons, I expect the field to keep rediscovering that supervised intuitions need to be re-derived from the actual structure of the RL objective rather than imported wholesale.