The Optimizer's Dilemma: How Adam and SGD Differ in Their Forgetting Dynamics

Introduction

Catastrophic forgetting remains one of the most persistent obstacles in the development of artificial neural networks capable of continual learning. Despite decades of research since McCloskey and Cohen first documented the phenomenon in 1989, our understanding of what precisely constitutes forgetting, and how to measure it, remains surprisingly fragmented. A recent study by Ashley, Ghiassian, and Sutton titled "Does the Adam Optimizer Exacerbate Catastrophic Forgetting?" brings this methodological ambiguity into sharp relief while uncovering a counterintuitive relationship between optimization algorithms and memory retention.

The paper addresses two fundamental questions that have received insufficient scrutiny in the continual learning literature. First, how should we quantify catastrophic forgetting in a way that captures its multifaceted nature? Second, to what extent do our implementation choices, specifically the selection of gradient-based optimizers, influence the severity of forgetting? Their findings suggest that much of the existing research may rest on shaky methodological ground, and that the widespread adoption of adaptive optimizers like Adam may inadvertently be undermining the very stability we seek in online learning systems.

The Metrics Problem: When Measurement Defines Reality

Perhaps the most unsettling contribution of Ashley et al. is their demonstration that the choice of evaluation metric can fundamentally alter the conclusions of a catastrophic forgetting study. The authors systematically compare four distinct metrics: retention, relearning, activation overlap, and pairwise interference. Each metric captures a different facet of the forgetting phenomenon, yet they often produce contradictory rankings when comparing optimization algorithms.

Retention measures the drop in performance on previously learned tasks after training on new tasks, without any re-exposure to the old data. This is the most common metric in continual learning benchmarks, typically calculated as the difference between final accuracy on task one and accuracy on task one after training on task n. Relearning, by contrast, measures how quickly performance on old tasks recovers when the model is briefly re-exposed to them, quantifying the residual knowledge that persists in the network weights despite apparent forgetting. Activation overlap examines the similarity of hidden layer representations across tasks, while pairwise interference analyzes how gradients from different tasks conflict during joint training.

The authors' empirical results reveal alarming sensitivity to these measurement choices. Under retention metrics, Adam often appears to suffer significantly more forgetting than vanilla SGD. However, under relearning metrics, the gap narrows or even reverses in some experimental conditions. This discrepancy is not merely academic; it suggests that Adam might be finding solutions that appear to forget under standard benchmarks but actually retain latent knowledge that can be rapidly reactivated. Conversely, SGD may appear more stable under retention metrics while actually overwriting useful features in a way that requires extensive retraining to recover.

This metric dependence creates a methodological crisis for the field. If the relative performance of algorithms shifts dramatically based on whether we measure retention or relearning, then claims about "state of the art" forgetting mitigation become largely meaningless without comprehensive evaluation. Ashley et al. argue convincingly that future work in supervised continual learning must report both retention and relearning metrics concurrently, while reinforcement learning studies focusing on intra-task forgetting should prioritize pairwise interference over activation overlap.

The SGD Advantage: Why Simplicity Resists Forgetting

The second major finding concerns the comparison between vanilla SGD and modern adaptive optimizers. Contrary to the assumption that algorithmic sophistication necessarily improves learning dynamics, the authors provide strong evidence that vanilla SGD consistently experiences less catastrophic forgetting than Adam across multiple testbeds, with particularly pronounced differences in reinforcement learning domains.

This result challenges the prevailing wisdom in deep learning, where Adam has become the default optimizer for most applications due to its rapid convergence and robustness to hyperparameter choices. Ashley et al. compare vanilla SGD, SGD with Momentum, RMSProp, and Adam across both supervised learning tasks involving sequential task learning and reinforcement learning scenarios involving non-stationary environments. Their results suggest that the adaptive learning rates and second-moment estimation that make Adam effective for rapid convergence may simultaneously increase plasticity at the cost of stability.

The mechanism behind this phenomenon likely relates to the geometry of the loss landscape. Adam's adaptive updates allow the optimizer to navigate narrow valleys and sharp minima effectively, but these precise solutions may be more brittle when the loss landscape shifts due to new tasks. SGD, with its noisier updates and implicit regularization properties, tends to find broader, flatter minima that are more robust to parameter perturbations. In the context of continual learning, these flat minima correspond to solutions that do not rely on finely tuned weight configurations, making them less susceptible to being overwritten by subsequent gradient updates.

Furthermore, SGD with Momentum occupies an intermediate position between vanilla SGD and Adam, suggesting that the accumulation of velocity vectors provides some of the benefits of adaptive optimization without the full destabilizing effects of per-parameter learning rates. RMSProp, which uses adaptive learning rates but lacks momentum, shows forgetting patterns closer to Adam, indicating that the adaptive scaling itself, rather than the momentum component, may be the primary driver of increased forgetting.

Broader Implications and Open Questions

These findings force a reconsideration of how we design continual learning systems. The community has focused extensively on architectural modifications, regularization terms, and memory replay mechanisms to combat forgetting, while largely treating the optimizer as a fixed hyperparameter to be tuned for convergence speed. Ashley et al. demonstrate that this separation is artificial; the optimizer is itself a critical component of the stability-plasticity dilemma.

The observation that SGD forgets less than Adam raises deeper questions about the nature of learning in neural networks. If adaptive optimizers are effectively overfitting to each task in sequence, finding sharp minima that encode task-specific information in fragile weight configurations, then the solution may not lie in adding complexity to prevent forgetting, but rather in embracing the implicit regularization of simpler optimization procedures. This connects to broader work on the implicit bias of gradient descent, suggesting that flat minima discovered by SGD naturally encode solutions that are robust to distribution shifts and task switches.

However, the trade-off cannot be ignored. While SGD may forget less, it typically converges more slowly and requires more careful learning rate scheduling. In resource-constrained environments or applications requiring rapid adaptation, the forgetting costs of Adam might be acceptable in exchange for faster initial learning. The paper's recommendation to use multiple metrics becomes crucial here; a system evaluated only on retention might reject Adam unnecessarily, while one evaluated only on relearning might overestimate its stability.

Looking forward, the field requires optimizers explicitly designed for the continual learning setting, rather than adapting stationary learning algorithms to non-stationary problems. Such optimizers might incorporate adaptive learning rates for within-task convergence while reverting to SGD-like updates during task transitions, or they might maintain task-specific curvature information to prevent interference. The work of Ashley et al. provides the methodological foundation for such developments by establishing that our choice of optimizer profoundly influences what we perceive as catastrophic forgetting, and that our perception depends critically on how we choose to measure it.