Model-Based Reinforcement Learning Under Non-Stationarity: When Data Expires

Introduction

Model-based reinforcement learning has emerged as the dominant paradigm for sample-efficient control in complex physical systems. By learning a predictive model of environment dynamics and using it for internal planning, these methods achieve remarkable data efficiency compared to their model-free counterparts. Yet most theoretical guarantees and practical implementations rest upon a fragile assumption: that the underlying system dynamics remain stationary throughout the learning process.

In their recent work, "Model-Based Reinforcement Learning for Control under Time-Varying Dynamics," Klemens Iten and colleagues from ETH Zurich directly challenge this assumption. The paper addresses a ubiquitous real-world phenomenon: robotic systems degrade through wear, vehicles encounter varying loads, and environmental conditions shift over time. When dynamics drift, historical data becomes stale, potentially biasing learned models and degrading control performance. The authors propose a rigorous framework for understanding this challenge and derive practical algorithms that adapt to temporal variation while maintaining theoretical guarantees.

The Variation-Budget Framework and Theoretical Foundations

The paper formalizes non-stationarity through an episodic setting where the true dynamics function $f^*_n$ evolves across episodes $n \in {1, \ldots, N}$. Rather than modeling this drift stochastically, the authors adopt a frequentist variation-budget assumption, which constrains the total variation in dynamics across the learning horizon. This assumption, common in the bandit literature but novel in continuous MBRL, posits that the cumulative change in the dynamics function remains bounded.

Using Gaussian process dynamics models, the authors analyze how this temporal drift affects learning guarantees. Their analysis yields a critical insight: to maintain calibrated uncertainty estimates and achieve sublinear dynamic regret, algorithms must explicitly limit the influence of outdated data. In stationary settings, MBRL benefits from accumulating all historical data to refine model estimates. Under non-stationarity, however, older samples drawn from $f^_{n-k}$ for large $k$ poison the model of current dynamics $f^_n$, creating a bias-variance trade-off that grows with the drift rate.

This theoretical result motivates the core algorithmic contribution: optimistic model-based RL with restricted data buffers. The authors propose two variants, R-OMBRL (periodic reset) and SW-OMBRL (sliding window). R-OMBRL periodically clears the data buffer entirely, ensuring that model training uses only recent trajectories. SW-OMBRL employs a sliding window mechanism that retains only the $w$ most recent episodes, discarding older data continuously. Both approaches sacrifice sample efficiency in stationary phases to maintain robustness during drift.

Practical Algorithms and Empirical Validation

Translating theoretical insights into scalable algorithms presents significant challenges. While the analysis assumes Gaussian process models for tractability, high-dimensional continuous control tasks require neural network function approximators. The authors bridge this gap by maintaining the core principle of optimism under uncertainty: they use the epistemic uncertainty of the learned dynamics model as an intrinsic reward signal to guide exploration.

In the proposed algorithms, a neural network ensemble or probabilistic network estimates both the expected next state and the uncertainty around that prediction. During planning, the agent optimizes a composite objective combining extrinsic task rewards and an uncertainty bonus. This encourages the agent to explore regions where the model is uncertain, crucial for rapidly adapting when dynamics shift.

The empirical evaluation focuses on continuous control benchmarks modified to exhibit time-varying dynamics. The results demonstrate that both R-OMBRL and SW-OMBRL outperform standard MBRL baselines that naively aggregate all historical data. As predicted by the theory, the performance gap widens as the drift rate increases. The sliding window variant generally achieves superior performance to periodic resets, suggesting that gradual data expiration preserves more useful information while still filtering out harmful stale samples.

Your Take: The Abrupt Change Challenge and Buffer Design

While the variation-budget framework provides elegant regret bounds, it encodes an assumption of gradual, bounded drift. The theory assumes that dynamics evolve continuously within a constrained total variation budget. This assumption, however, may fail to capture critical real-world scenarios: a robot joint seizes mid-operation, a tire bursts changing vehicle dynamics instantaneously, or a manufacturing tool fractures. These abrupt change points violate the smooth drift model and present a fundamental limitation to fixed-window approaches.

Consider the behavior of SW-OMBRL when faced with a sudden discontinuity. If the window size $w$ is tuned for gradual drift, the buffer will retain substantial pre-change data, polluting the model until those samples age out naturally. Conversely, shrinking $w$ to handle abrupt changes increases variance during stable periods and reduces sample efficiency when the system is stationary. This reveals a deeper tension: window size alone is insufficient for robust adaptation to heterogeneous change modes.

The path forward likely requires coupling change-point detection directly into the buffer mechanism. Rather than passively expiring data based on age, adaptive algorithms should actively detect distribution shifts in the incoming trajectory data. Recent advances in Bayesian surprise measures and predictive likelihood monitoring could trigger buffer clears or weighting adjustments when statistically significant shifts occur. This represents a shift from time-based forgetting to event-based forgetting, aligning the learning system with the causal structure of physical wear and failure modes.

Furthermore, the current framework treats all dimensions of the state-action space equally. In reality, physical systems often exhibit localized non-stationarity: a single actuator degrades while others remain stable, or a specific operational regime changes while others persist. Future algorithms might employ component-specific buffers, maintaining separate data retention policies for different subspaces of the dynamics. This granular approach would preserve valuable data in stable dimensions while rapidly adapting in varying ones.

Conclusion

The work by Iten et al. establishes a rigorous foundation for MBRL under non-stationarity, proving that limiting stale data is not merely a heuristic but a theoretical necessity for sublinear regret. By connecting variation-budget analysis to practical buffer mechanisms, they provide a template for building adaptive control systems that resist the entropic decay of outdated information.

Yet significant challenges remain. The field must move beyond uniform windowing strategies toward intelligent, context-aware data selection. Integrating explicit change-point detection, hierarchical buffer structures, and perhaps meta-learning approaches that adapt the adaptation rate itself represents the next frontier. As robotic systems operate in increasingly unstructured environments over extended durations, the ability to distinguish between relevant experience and historical artifact will separate robust autonomous systems from those trapped by their own outdated models.