Forgetting as Routing: Rethinking Continual Reinforcement Learning

When we talk about catastrophic forgetting in continual learning, the conversation almost always slides toward memory. The intuition is seductive: an agent forgets a past task because it no longer holds the relevant parameters, so the solution must be to store more of them. Replay buffers, parameter masks, growing networks, and expanding latent pools all follow this logic. But I want to argue, with the help of a recent paper, that this framing is incomplete. The deeper problem in continual reinforcement learning is not how much you remember. It is how cleanly you separate what conflicts from what aligns.

The paper in question is Continual Knowledge Adaptation for Reinforcement Learning (CKA-RL). It proposes a method that maintains a task-specific knowledge vector pool and dynamically adapts historical knowledge to new tasks, paired with an adaptive merging mechanism that compresses redundant knowledge. The reported gains are a 4.20% improvement in overall performance and an 8.02% improvement in forward transfer across three benchmarks. Those numbers are modest in absolute terms, but the mechanism behind them is what I find worth examining.

The Standard Framing and Why It Misleads

Continual reinforcement learning sits at an uncomfortable intersection. Standard RL already struggles with non-stationarity inside a single task because the policy changes the data distribution as it learns. Continual RL adds a second axis of non-stationarity: the task itself changes over time. An agent trained on a sequence of locomotion tasks or manipulation goals must absorb new objectives without destroying the competence it built earlier.

The dominant response has been to treat this as a storage and protection problem. Methods like Elastic Weight Consolidation protect important weights with quadratic penalties. Progressive networks add new columns of parameters per task. Replay-based methods keep samples or generative models of past experience. All of these implicitly assume that forgetting happens because the relevant information is overwritten or lost.

The trouble is that storage-centric methods scale poorly and, more importantly, they do not address why gradient updates from a new task degrade old behavior in the first place. The mechanism is interference: the gradients that improve task B push parameters in directions that harm task A, because the two tasks share representation capacity that encodes conflicting demands. Adding more parameters reduces the chance of collision, but it does not give you a principled account of which knowledge conflicts and which can be safely shared.

What CKA-RL Actually Does

CKA-RL reframes the problem around knowledge vectors rather than raw experience. The core ideas are two.

• Continual Knowledge Adaptation. The method maintains a pool of task-specific knowledge vectors, where each vector encodes a compact representation of the parameter adjustments associated with a task. When a new task arrives, the agent does not learn from scratch nor blindly fine-tune. It selects and adapts relevant historical knowledge to bootstrap the new policy. This preserves critical parameters that worked before while still allowing adaptation.

• Adaptive Knowledge Merging. Because a growing pool of per-task vectors would eventually become a storage liability, CKA-RL merges similar knowledge vectors. Vectors that are close in the relevant space get consolidated, reducing memory while retaining the essential structure.

This is where the interpretation matters. On the surface, merging looks like a compression trick to keep the buffer small. And the paper presents it partly that way, as a scalability solution. But notice what merging actually encodes: a judgment about which pieces of knowledge are redundant or aligned, and therefore safe to combine, versus which must be kept separate because they encode distinct or conflicting task structure.

That is not really a memory operation. It is a routing operation. The system is deciding which knowledge participates in which task, and how strongly. The 8.02% forward transfer gain, which is the largest improvement reported, supports this reading. Forward transfer measures how much learning previous tasks helps with future ones. If the win were purely about remembering more, you would expect the gains concentrated in backward metrics, that is, in not forgetting old tasks. The fact that the strongest effect appears in forward transfer suggests the method is succeeding at reusing aligned knowledge for new problems, which is fundamentally a question of selection and combination, not retention.

Forgetting as Bad Routing

Here is the inference I want to push. In a parameter-sharing network trained sequentially, forgetting is frequently a symptom of poor credit assignment across tasks. The optimizer cannot tell which weights carry task-specific knowledge that should be insulated and which carry transferable knowledge that should be updated freely. So it updates everything, and old behavior degrades.

A knowledge vector pool with adaptive merging is, in effect, a structured routing prior. It says: keep distinct task signatures separate when they would interfere, and tie them together when they agree. Viewed this way, the method is solving an attribution problem. It is deciding where each task's learning signal should land. Compression is a side effect of doing this attribution well, because well-attributed knowledge is naturally low-rank across similar tasks.

This connects to a broader theme in the field. Parameter-efficient adaptation methods such as low-rank adapters, and mixture-of-experts routing in large models, all share the same underlying insight: capacity should be allocated conditionally, not uniformly. CKA-RL is bringing a version of that conditional allocation into the continual RL setting, where the conditioning variable is the task identity inferred through knowledge similarity. The line between a knowledge vector pool and a set of task-conditioned experts is thinner than it first appears.

Limitations and Honest Caveats

Several things deserve scrutiny. First, the absolute improvement of 4.20% in overall performance is incremental. Continual RL benchmarks are notoriously sensitive to task ordering, task similarity, and hyperparameter tuning, so the gain needs to be read against the variance of these benchmarks before treating it as decisive.

Second, the merging mechanism depends on a similarity measure. The quality of the entire approach hinges on whether that similarity actually tracks functional interference. Two knowledge vectors might look close geometrically yet encode behaviors that conflict in deployment, or look distant yet be safely shareable. The paper would need to demonstrate that its similarity criterion aligns with true task interference, not just representational proximity.

Third, the routing-versus-memory distinction I am drawing is partly interpretive. The paper frames merging as a scalability solution. My claim that the real benefit is cleaner credit assignment is an inference from the metric profile, not something the authors explicitly isolate through ablation. A targeted experiment, measuring interference directly before and after merging, would be needed to confirm it.

Closing Thoughts

If forgetting is often bad routing in disguise, then the research agenda for continual RL should shift accordingly. Instead of asking how to store more, we should ask how to attribute learning signals to the right capacity, and how to detect and isolate interference early. CKA-RL is a useful data point in that direction because its strongest result is in transfer, the metric most sensitive to good knowledge reuse.

The open questions I find most interesting are these. Can we design similarity measures that are explicitly grounded in gradient interference rather than representational distance? Can knowledge merging be made adaptive in both directions, splitting a merged vector when a new task reveals a latent conflict? And at scale, does this framing converge with the routing machinery already used in large conditional models, suggesting one unified theory of capacity allocation across both supervised and reinforcement settings? Those are the threads worth pulling next.