Bayesian Online Meta-Learning: A Principled Approach to Sequential Few-Shot Learning

Neural networks, despite their remarkable capacity for pattern recognition, suffer from a fundamental vulnerability known as catastrophic forgetting. When trained sequentially on new tasks, these models tend to overwrite previously learned knowledge, a phenomenon well documented in large scale supervised classification. However, the intersection of catastrophic forgetting with few shot learning remains relatively unexplored territory. The paper "Addressing Catastrophic Forgetting in Few-Shot Problems" identifies a critical gap in the literature, demonstrating that Model Agnostic Meta Learning (MAML), the dominant gradient based meta learning algorithm, fails catastrophically when confronted with sequentially arriving few shot tasks. The authors propose a Bayesian online meta learning framework that fundamentally reimagines how meta parameters should be updated in continual learning scenarios.

The Plasticity Trap: Why MAML Fails at Continual Few-Shot Learning

MAML operates on an elegant premise: find an initialization such that a small number of gradient steps on a new task produces effective task specific parameters. This meta objective has proven remarkably effective for stationary few shot learning, where the model encounters training tasks drawn from a fixed distribution before deployment. However, when tasks arrive sequentially during meta training itself, MAML exhibits severe forgetting.

The root cause lies in MAML's treatment of meta parameters as uniformly plastic. During meta updates, MAML computes gradients across tasks and applies uniform updates to all parameters, lacking any mechanism to distinguish between weights critical for previously learned task structures and those available for new learning. Without uncertainty tracking, the algorithm blindly overwrites parameters that encoded essential inductive biases from earlier tasks. This creates a stability plasticity dilemma at the meta level; the model must remain adaptable enough to learn new few shot tasks while preserving the meta knowledge that enables rapid adaptation in the first place.

In standard supervised continual learning, one might employ replay buffers or parameter regularization to mitigate forgetting. Few shot learning presents unique constraints. The very definition of the problem precludes storing large buffers of previous task data, and the meta learning process involves bilevel optimization that complicates traditional regularization approaches. When MAML encounters task two after meta learning on task one, the meta gradient update for task two does not account for how these changes will affect the loss landscape of task one. Compounded across hundreds of sequential tasks, this leads to complete erasure of earlier meta knowledge.

The Bayesian Solution: Uncertainty Aware Meta Learning

The proposed framework addresses this limitation by treating meta learning as Bayesian online inference. Rather than maintaining point estimates of meta parameters, the framework maintains a posterior distribution over these parameters, quantifying uncertainty about which values best support rapid adaptation across all observed tasks.

The technical implementation combines Laplace approximation with variational inference to render this approach computationally tractable. Laplace approximation assumes the posterior distribution over meta parameters is approximately Gaussian centered at the maximum a posteriori estimate. This allows the framework to track not just the mean parameter values but also the curvature of the loss landscape, encoded in the precision matrix (inverse covariance). Parameters associated with high curvature, indicating they are well determined by previous tasks and critical for maintaining performance, receive strong regularization against change. Parameters with low curvature, suggesting they remain underdetermined or less critical, retain flexibility for new learning.

Variational inference provides the computational machinery to approximate the posterior without requiring expensive MCMC sampling. By optimizing a variational lower bound on the marginal likelihood, the framework can meta learn on sequentially arriving tasks while maintaining an approximation of the full posterior. This creates a natural mechanism for determining which parameters should remain frozen and which should stay adaptable, based on accumulated uncertainty estimates from the history of observed tasks.

Notably, the authors demonstrate an additional utility beyond pure continual learning. The framework can meta learn on sequentially arriving few shot tasks even when drawn from a stationary task distribution, suggesting that the Bayesian treatment provides benefits for standard meta learning scenarios beyond just preventing forgetting.

Broader Implications and Technical Considerations

This work sits at the intersection of several active research threads in machine learning. It connects meta learning with continual learning through the lens of Bayesian neural networks, offering a principled alternative to heuristic approaches like Elastic Weight Consolidation (EWC) or Synaptic Intelligence (SI). While EWC approximates parameter importance using the Fisher information after training on each task, the Bayesian online framework integrates uncertainty estimation directly into the meta learning process itself. This distinction matters because in few shot learning, the limited data per task makes post hoc importance estimation particularly noisy and unreliable.

The framework also relates to recent advances in uncertainty quantification for neural networks. By maintaining a posterior over meta parameters rather than task specific parameters, the method implicitly captures uncertainty about the learning process itself, not just predictions. This meta level uncertainty proves crucial for determining how much a neural network architecture can safely learn without destroying its existing capabilities.

However, the approach carries significant computational overhead. Maintaining a full Laplace approximation requires storing and updating a covariance matrix quadratic in the number of meta parameters. For modern neural architectures with millions of parameters, this remains prohibitive. The authors acknowledge this limitation implicitly by employing variational methods, though the paper leaves open questions regarding scalability to large scale architectures like deep residual networks or transformers.

Furthermore, the theoretical guarantees assume specific conditions about task distributions. While the paper demonstrates efficacy on sequentially arriving tasks from stationary distributions, real world continual learning often involves non stationary distributions where task characteristics drift over time. The Laplace approximation also assumes local Gaussianity of the posterior, an assumption known to break down in deep networks with complex loss landscapes featuring multiple modes and flat regions.

Looking Forward: Personalized Edge AI and Beyond

The implications of this work extend far beyond academic benchmarks. We are witnessing convergence toward personalized edge AI systems, smartphones, robots, and wearables that must learn user specific tasks from handfuls of examples without forgetting previously acquired skills. Current MAML based approaches deployed in such settings would fail as users introduce new tasks sequentially; a robot learning to recognize new objects in a home would progressively forget previously learned categories.

The Bayesian framework offers a principled foundation for such applications. By quantifying parameter importance via variational bounds, devices can make informed decisions about which weights to update during on device personalization. The computational constraints that currently limit deployment will likely ease through sparse Laplace approximations, where only blocks of the precision matrix are maintained, or through low rank approximations of the posterior covariance.

Several open questions remain. How does this framework interact with neural architecture search, where the model structure itself might adapt to new tasks? Can the Bayesian treatment extend to meta reinforcement learning, where environments rather than classification tasks arrive sequentially? Most importantly, how do we relax the stationarity assumptions to handle true lifelong learning where the task distribution evolves in structured ways?

The paper "Addressing Catastrophic Forgetting in Few-Shot Problems" establishes a rigorous foundation for these investigations. By bringing Bayesian online learning to meta learning, the authors transform catastrophic forgetting from an accepted limitation into a solvable inference problem. As we build machines intended to learn continuously alongside humans, this principled approach to preserving knowledge while maintaining adaptability will prove essential.