Reset-Free Lifelong Learning with Skill-Space Planning: Escaping the Episode Reset Trap

Reinforcement learning in the laboratory operates under a convenient fiction. When an agent fails, we reset the environment. When the task changes, we reinitialize. This episodic framing has produced remarkable successes in games and robotics, yet it masks a fundamental brittleness. The real world offers no reset button. Autonomous systems must operate continuously, recovering from arbitrary states while adapting to environmental changes that never pause for training.

The paper Reset-Free Lifelong Learning with Skill-Space Planning addresses this gap directly. The authors propose Lifelong Skill Planning (LiSP), a framework that abandons the episodic assumption entirely. By operating in a learned space of reusable skills rather than raw actions, LiSP enables agents to plan over long horizons, recover from catastrophic situations, and adapt to non-stationary dynamics without requiring environment resets. This work represents a necessary evolution toward the kind of continuous, autonomous operation that deployed systems actually require.

The Non-Episodic Reality

Standard reinforcement learning assumes an episodic structure. Episodes begin at sampled initial states, proceed according to the agent's policy, and terminate, at which point the environment resets. This assumption simplifies credit assignment and exploration, but it collapses under the weight of continuous deployment. In non-episodic settings, the agent must handle arbitrary initial states, including those that represent catastrophic failures or distribution shifts. Without the ability to reset, standard policy gradient methods and value-based approaches suffer from compounding errors. The state distribution becomes non-stationary not merely because the environment changes, but because the agent's own mistakes can drive it into regions from which recovery is impossible.

LiSP confronts this by recognizing that planning at the level of individual actions is computationally intractable for long horizons and brittle under perturbation. Instead, the framework operates at a higher level of abstraction.

Architecture of Skill-Space Planning

The core insight of LiSP lies in decoupling skill acquisition from skill sequencing. The framework consists of three integrated components: unsupervised skill discovery, a learned dynamics model in skill space, and a planning module.

First, skills are discovered without external reward signals using intrinsic motivation. The agent learns a set of reusable behaviors that partition the state space into distinguishable outcomes. This unsupervised approach is critical because it allows the agent to build a behavioral repertoire before encountering specific tasks or during offline phases using previously collected data. The paper emphasizes this offline capability, noting that skill discovery from static datasets reduces the need for dangerous or costly real-world interactions during learning.

Second, LiSP learns a dynamics model not over raw states, but over the skill space itself. This temporal abstraction compresses long trajectories into single transitions between skills. Planning in this compressed space is computationally efficient and naturally handles long-horizon reasoning. The planner can sequence skills to achieve goals while ignoring low-level motor noise.

Third, the framework enables recovery from arbitrary states. Because skills are reusable and the planner operates over the skill space, the agent can generate a sequence of skills to return to viable states even when current conditions are dire. This contrasts sharply with standard RL, where recovery policies are rarely learned explicitly.

Empirical Validation in Non-Stationary Domains

The authors evaluate LiSP on both discrete gridworld environments and continuous MuJoCo control tasks, introducing non-stationarity through changing dynamics, shifting goals, or environmental obstacles that appear over time. The results demonstrate that standard flat RL methods fail catastrophically in these conditions, often becoming stuck in irrecoverable states or failing to adapt to new dynamics.

LiSP succeeds by maintaining a stable set of skills while adapting the high-level plan. In gridworlds with changing walls and goal locations, the agent reuses navigation skills while replanning the route. In MuJoCo locomotion tasks where terrain properties change continuously, the skill space provides robustness; individual skills remain viable even as their optimal sequencing changes.

Particularly notable is the framework's ability to avoid catastrophic failures. By planning in skill space, the agent maintains awareness of reachable states and can avoid actions that lead to