Lifelong Reinforcement Learning: Why the Reset Button Breaks the Theory

Introduction

Most of the reinforcement learning we discuss in research papers and deployment write-ups shares a hidden assumption so deeply embedded that we rarely question it: the episode. We define a task, the agent interacts with the environment until some terminal condition, we tally up the reward, and then we reset. The world snaps back to a starting distribution, and the agent tries again. This convenient fiction underpins much of our theory, from the convergence guarantees of value iteration to the way we benchmark agents on Atari and MuJoCo.

The paper "Some Insights into Lifelong Reinforcement Learning Systems" by Changjian Li argues that this fiction, while useful, fundamentally misrepresents the kind of learning we ultimately care about. A genuinely lifelong agent has no reset button. It lives one long trajectory, and there is no second chance to run the experiment again. In my recent post I framed this as a homeostasis problem borrowed from biology. Here I want to work through the argument more carefully, examine the prototype the paper proposes, and lay out where I think the implications lead.

Why the Episodic Paradigm Fails

The standard formulation of reinforcement learning rests on the Markov Decision Process, where an agent seeks a policy that maximizes expected cumulative discounted reward. The objective is well posed because we assume access to many episodes drawn from the same distribution. Over repeated trials, the agent can estimate value functions, reduce variance, and converge toward an optimal policy. The mathematics is clean precisely because the environment is, in effect, repeatable.

The paper's central observation is that lifelong learning violates the conditions that make this objective coherent. Consider the difficulties:

• No repeated trials. In a lifelong setting the agent experiences a single, non-repeating stream of interaction. There is no ensemble of episodes over which to define an expectation in the usual sense.
• No fixed optimal policy. If the environment is non-stationary, the notion of converging to a single optimal policy becomes ill-defined. What is optimal at one point in the agent's life may be harmful later.
• Irreversibility. Some actions terminate the learning process permanently. An episodic agent that "dies" simply starts a new episode; a lifelong agent that dies is finished. This asymmetry is not a minor detail. It changes the structure of the problem.

The paper presses on the question of what an objective function even means in this context. In episodic RL we maximize an expectation over a distribution of trajectories. But if the agent only ever produces one trajectory, the expectation is being taken over something the agent never actually samples more than once. We end up optimizing a quantity that has no operational meaning to the agent living its single life.

This is where I find the biological framing most useful. An organism does not maximize a scalar return computed over hypothetical re-runs of its existence. It maintains the internal and external conditions necessary to keep functioning. The relevant criterion is not "did you achieve the highest possible cumulative reward" but "did you stay alive and viable as conditions shifted."

The Prototype and the Shift to Viability

The paper offers a deliberately simplistic prototype lifelong reinforcement learning system to make the abstract argument concrete. The value of such a prototype is not that it outperforms existing agents on a benchmark; it is that it embodies a different design philosophy. The agent is constructed to operate over an unbounded interaction stream, adapting continually rather than driving toward a converged policy.

Several features distinguish this kind of system from the standard setup:

• Continual adaptation as the default mode. Rather than treating learning as a phase that precedes deployment, the system treats learning as a permanent condition. There is no clean separation between training and test.
• Local rather than global optimality. The agent is concerned with behaving acceptably given current conditions, not with provably optimal long-horizon behavior across the whole lifetime.
• Sensitivity to irreversible failure. Because there is no reset, avoiding catastrophic states carries a weight that a discounted reward sum does not naturally capture.

I want to develop the viability framing further than the paper does, because I think it offers a cleaner mathematical alternative. In dynamical systems and control theory there is a well-developed notion of a viability set: the set of states from which the system can, with some admissible policy, remain within a feasible region indefinitely. The control objective is not to reach a target or maximize a return but to keep the trajectory inside the viability kernel. This reframes the problem from optimization toward constraint satisfaction over an infinite horizon.

Under this view, the lifelong agent's job is to stay within the viability set despite a drifting environment. Reward, if it appears at all, becomes a secondary signal that shapes preferences among viable behaviors rather than the primary thing being maximized. Survival, in my framing, acts as a regularizer: it bounds the space of acceptable policies before reward ever enters the picture.

My Take: Different Math, Not Just Harder RL

A common reaction to lifelong learning is to treat it as ordinary RL plus extra engineering. Add replay buffers to fight forgetting, add meta-learning to adapt faster, add intrinsic motivation to explore. These are valuable techniques, but they leave the underlying objective untouched. The paper's deeper point, as I read it, is that the objective itself is the problem.

If we take the viability perspective seriously, several things follow:

• The discount factor loses its privileged role. Discounting is partly a mathematical convenience to keep infinite sums finite. In a viability framing, the infinite horizon is handled by the requirement to remain in a feasible set, not by geometric decay of future reward.
• Risk asymmetry becomes structural. A viability-based agent inherently treats irreversible failure differently from recoverable losses, because leaving the viability set is terminal by definition. Standard return maximization only captures this if we hand-engineer the reward to do so.
• Evaluation must change. We cannot meaningfully report "average return over episodes" for a system with no episodes. The right metrics concern how long the agent remains viable, how it responds to distribution shift, and whether it avoids absorbing failure states.

I also see a connection to multi-objective and constrained RL, and to the safe RL literature where constraints on visitation of dangerous states are added to a reward objective. The viability view suggests that for lifelong agents, the constraint is not an add-on but the primary structure, with reward demoted to a tie-breaker.

Limitations and Honest Caveats

I should be clear about what this line of argument does not yet deliver. The paper is conceptual and its prototype is, by its own description, simplistic. There is no large-scale empirical demonstration that a viability-first agent outperforms a well-tuned episodic agent on any realistic task. Defining the viability set itself is hard; in most interesting environments we do not know the boundary of the feasible region in advance, and learning it is its own difficult problem.

There is also a risk of overstating the dichotomy. Many practical systems sit between pure episodic and pure lifelong regimes. A recommender system serving a non-stationary user population, for example, has soft resets and partial repeatability. The episodic abstraction is wrong in the limit but often serviceable in practice. We should resist the temptation to discard a useful approximation simply because it is not exact.

Conclusion

The paper makes a quiet but important point: the episodic assumption is not a neutral modeling choice. It shapes what we optimize, what we can prove, and ultimately what kind of intelligence we build. A lifelong agent in a non-stationary world is not solving for a fixed optimal policy, because no such fixed object exists. It is maintaining viability across shifting conditions, which is a different goal requiring different mathematics.

The open questions I find most pressing are these. Can we formulate a tractable learning objective grounded in viability rather than return, and derive learning algorithms for it that scale? How do we learn the boundary of the viability set online, from a single irreversible trajectory, without dying in the process? And can reward maximization be recovered as a special case of a more general viability-based theory, or are they genuinely distinct? Answering these would move lifelong reinforcement learning from a critique of the standard paradigm toward a constructive alternative.