The Category Error at the Heart of Continual Reinforcement Learning

Introduction

The fundamental premise of classical reinforcement learning contains a hidden teleology. The agent seeks an optimal policy; once found, learning terminates. This framework, elegant and mathematically tractable, has driven decades of progress from Atari game playing to robotic control. Yet it stands in direct tension with the emerging paradigm of continual reinforcement learning, where environments evolve, tasks accumulate, and learning persists indefinitely without the comfort of a final state or an eventual reset.

In their paper "Rethinking the Foundations for Continual Reinforcement Learning," the authors articulate a profound structural mismatch between the tools we use and the problems we claim to solve. They argue that continual reinforcement learning has been hobbled by its reliance on foundations designed for a fundamentally different kind of problem. Specifically, they identify four pillars of traditional reinforcement learning that actively undermine the goals of continual adaptation: the Markov decision process formalism, the obsession with atemporal artifacts, the expected sum of rewards as an evaluation criterion, and episodic benchmark environments that instantiate the previous three assumptions.

This is not merely a technical quibble about nonstationarity. It is a categorical error in how