Composition as Continual Learning: A Look at Self-Composing Policies

Introduction

Continual reinforcement learning sits at an uncomfortable intersection of two desirable but often conflicting properties: the ability to retain what was learned on previous tasks, and the ability to keep learning new ones without degradation. The standard failure mode, catastrophic forgetting, has been studied for decades, yet the reinforcement learning setting compounds the difficulty. Unlike supervised learning, the data distribution in RL is non-stationary even within a single task, and the policy itself shapes the data it sees.

The paper "Self-Composing Policies for Scalable Continual Reinforcement Learning" proposes a structural answer rather than a regularization-based one. Instead of penalizing changes to important weights (as in elastic weight consolidation) or replaying old experiences, it builds a network that grows by adding modules, where each new module can selectively compose the policies learned by its predecessors. In this article I want to examine what makes this approach attractive, where its scaling claims hold up, and where I think the harder questions begin.

The Architecture and Its Claims

The central idea is a growable, modular network. When a new task arrives, a new module is instantiated. Crucially, this module does not learn in isolation. It can combine the outputs of previously learned policies along with its own internal policy. This composition is the mechanism by which knowledge transfer happens: if a prior task taught a useful sub-behavior, the new module can route through it instead of relearning it from scratch.

Three properties are claimed, and they are worth separating carefully:

• No catastrophic forgetting. Because previously trained modules are frozen, their policies are preserved exactly. This is the cleanest possible guarantee against forgetting. Nothing about training task N can corrupt the parameters responsible for task N minus 1. This is the same structural insight underlying progressive neural networks, but with a different compositional mechanism.

• Linear parameter growth. Earlier growing-network methods, most notably progressive networks, suffered from parameter growth that scaled poorly because each new column added lateral connections to all previous columns, producing a quadratic blowup. The paper argues that its modular composition keeps total parameter count linear in the number of tasks. This is a meaningful improvement and is the headline scalability result.

• Preserved plasticity. Methods that protect old knowledge often do so by constraining the network, which reduces its capacity to absorb new tasks. The authors claim their approach does not trade plasticity for stability, because each new task gets fresh capacity in the form of a new module.

The experiments span continuous control benchmarks and visual problems, and the reported result is that the method achieves greater forward transfer and higher final performance than competing continual RL baselines. The combination of frozen modules (stability) plus fresh per-task modules (plasticity) plus composition (transfer) is the conceptual package being sold, and on its own terms it is coherent.

Why the Composition Mechanism Matters

The interesting part is not that modules are frozen; freezing is an old idea. The interesting part is how a new module accesses old ones. If the only way to reuse prior knowledge were to copy entire previous policies, the system would be brittle. The selective combination mechanism allows the new module to weight the contributions of prior policies, effectively learning a routing or attention-like distribution over the existing library.

This is where the design becomes genuinely elegant. Consider a sequence of locomotion tasks where each new gait shares some primitives with earlier ones. A composition mechanism lets the new module borrow the relevant primitive policies and spend its own capacity only on the residual difference. This is closely related to the idea of policy reuse and to options frameworks in hierarchical RL, but here the reuse is learned end to end and at the level of full policy modules rather than hand-specified options.

It is also worth connecting this to the broader trend toward modularity in deep learning, including mixture-of-experts systems. The shared intuition is that capacity should be conditionally activated rather than uniformly applied. In continual learning, the conditioning is on task identity, which makes the routing problem simpler than the input-conditioned routing of MoE models, but it introduces the assumption that task boundaries are known, a point I return to below.

My Take: The Scaling Story Is More Subtle Than Linear

I want to push on the scalability claim, because I think the word "linear" does real rhetorical work that deserves scrutiny.

First, linear parameter growth is still unbounded growth. Linear is better than quadratic, but it is not constant. After thousands of tasks the model becomes large regardless of the favorable constant factor. For a true lifelong learning system, the right asymptotic target is not a better growth rate but eventual consolidation, where redundant modules are merged or distilled so the total footprint plateaus. The paper improves the slope of the line; it does not change the fact that the line keeps rising. This is the single most important caveat for anyone imagining deployment over very long horizons.

Second, and more subtly, I suspect linear parameter growth can hide superlinear compute growth at inference. If each new module composes over all of its predecessors, then producing an action for task N may require evaluating or attending over a set of modules whose size grows with N. Parameter count and per-forward-pass cost are not the same quantity. A library of M modules where the active module reads from all M predecessors implies an inference cost that scales with M even if the stored parameters scale linearly as well. The paper's framing emphasizes parameters; the practical bottleneck for a deployed agent may instead be latency and memory bandwidth during the composition step. I would want to see explicit wall-clock and FLOP measurements as the task count grows, not just parameter counts.

Third, there is the selection mechanism itself. Composition presupposes that the right prior policies can be identified and weighted appropriately. When the module library is small, learning these weights is easy. As the library grows, the routing problem becomes a search over an increasingly large space, and the gradient signal for selecting among many similar candidates may become noisy. I expect a regime where the selection mechanism, not the policy modules, becomes the limiting factor. This mirrors a known difficulty in large mixture-of-experts systems, where router collapse and load imbalance are persistent engineering problems. The continual RL version may be milder because task identity helps, but it does not vanish.

A further limitation worth naming honestly is the dependence on task boundaries. Most modular growing approaches, this one likely included, assume the agent knows when a new task begins so it can instantiate a new module. Task-agnostic continual learning, where boundaries are unobserved or blurred, is harder and is where many real applications actually live.

Conclusion and Open Questions

"Self-Composing Policies for Scalable Continual Reinforcement Learning" makes a clean and useful contribution. By combining frozen modules for stability, fresh modules for plasticity, and learned composition for transfer, it achieves strong continual RL performance while improving on the parameter scaling of earlier growing-network methods. The structural guarantee against forgetting is its most attractive feature, and the empirical transfer results suggest the composition mechanism is doing real work.

The questions I am left with are mostly about the long-horizon regime that the word "scalable" invites us to imagine:

• Can modules be consolidated or distilled so the footprint eventually plateaus rather than growing forever?
• What is the true inference cost as a function of task count, and does composition over all predecessors introduce a hidden superlinear term?
• At what library size does the selection mechanism become the bottleneck, and can task-similarity structure be exploited to keep routing tractable?
• How does the method degrade when task boundaries are unknown?

These are not criticisms of what the paper accomplishes; they are the natural next problems that its success exposes. Modular composition is a promising direction precisely because it reframes continual learning as a problem of reuse and routing rather than of weight protection. The frontier now is making that reuse cheap, bounded, and robust at the scale where lifelong learning actually becomes interesting.