Two Timescales of Learning: Why LLMs Should Adapt in Context and in Weights

Introduction

There is a recurring tension in how we adapt large language models to new tasks. On one side sits parameter updating: supervised finetuning, preference optimization, and reinforcement learning with verifiable rewards (RLVR). These methods are powerful but expensive, and they write every lesson, durable or transient, into the same set of weights. On the other side sits in-context adaptation: prompts, instructions, and scaffolds that can be edited cheaply and applied per task, but which typically cannot match the performance ceiling of weight updates on their own.

The paper "Learning, Fast and Slow: Towards LLMs That Adapt Continually" by Tiwari, Sareen, Agrawal and colleagues argues that this is a false dichotomy. Rather than choosing between learning in weights or learning in context, they propose learning in both simultaneously, with the two channels operating at different timescales. The framing borrows from the System 1 versus System 2 distinction and from a long lineage of fast-weight and slow-weight work in neural networks. The result is a training method they call Fast-Slow Training (FST), and its empirical consequences are more interesting than the headline efficiency number suggests.

The Method: Two Channels, Two Reward Signals

The core architectural idea is clean. FST treats LLM adaptation as joint optimization over two components:

• Slow weights (θ): the model parameters, expensive to update, persistent across tasks, encoding long-lived reasoning behavior.
• Fast weights (ϕ): a textual context pool drawn from discrete text space, cheap to edit, applied per task, capturing transient task-specific adaptation.

Given a query x, the system samples a response from the policy π_θ(· | x, ϕ), and the joint objective maximizes expected reward over both θ and ϕ. What makes the instantiation work in an RLVR setting is the way each channel is updated and, crucially, what information each channel is allowed to consume.

The slow loop runs reinforcement learning (PPO-style RLVR) using the scalar reward alone. The fast loop uses reflective context optimization via GEPA, and it consumes the full text of the rollout: thoughts, tool calls, error traces, and rich textual feedback. So a ZeroDivisionError on line 5, which collapses to a scalar reward of 0.0 for the slow update, becomes a detailed, actionable lesson for the fast update.

The two loops are interleaved so that context and policy co-evolve. This contrasts with the conventional pipeline of training a policy first and then optimizing a prompt for the final checkpoint. A further design choice is that the fast context is maintained as a Pareto-frontier population rather than a single best prompt. Different contexts specialize to different problem slices, which preserves diversity and exposes the slow update rule to varied conditioning during training. This is a subtle but important point: the slow learner is not optimizing against a fixed prompt but against a moving, diverse distribution of contexts, which likely acts as a form of regularization.

The Results, and Which One Actually Matters

The paper reports four empirical advantages across math, code, and general reasoning tasks:

1. Sample efficiency: FST matches RL reward with up to 3x fewer rollouts and reaches a higher performance asymptote.
2. Smaller weight displacement: at matched reward, FST models stay up to 70% closer in KL divergence to the base policy than RL-only baselines.
3. Preserved plasticity: after training on one task, FST checkpoints adapt effectively to a second task, while RL-only checkpoints collapse to near 0%.
4. Continual learning: when task domains change on the fly, FST keeps acquiring new tasks while parameter-only RL stalls.

The 3x sample efficiency is the number that draws attention, and it is genuinely useful given how rollout-bound RLVR pipelines are. But I think the more consequential result is the connection between low KL drift and preserved plasticity, which is what I emphasized in my earlier note on this work.

Here is the logic. Catastrophic forgetting and plasticity loss have generally been treated as distinct failure modes in the continual learning literature. Forgetting is about losing previously acquired knowledge; plasticity loss is about losing the capacity to acquire future knowledge. FST's results suggest these are two symptoms of a single underlying cause: weights absorbing information that a textual channel could have held instead. When the only adaptation channel is the parameters, every transient lesson, every task-specific heuristic, every reward-hacking shortcut gets compiled into θ. This drags the model away from base behavior, reduces output entropy, and saturates the parameter directions that future learning would need. By offloading the transient, task-specific signal into ϕ, FST keeps θ near the base model, and a model that has not over-committed its parameters retains the entropy and the slack required to learn the next thing.

My Take: A Division of Labor by Information Type

What I find most compelling here is not the specific GEPA-plus-RLVR instantiation, but the principle it exposes. The two channels are differentiated not just by timescale but by the granularity of feedback they can metabolize. A scalar reward is an extraordinarily lossy signal; it tells you that something went wrong without telling you what. Gradient descent through that scalar forces the model to attribute credit across billions of parameters from a single number. Text, by contrast, can carry causal, structured information: this specific line raised this specific exception. The fast channel is, in effect, a higher-bandwidth pathway for exactly the kind of lessons that are easy to state in language and wasteful to encode in weights.

This reframes a familiar intuition. We often say that weights should hold general skills and prompts should hold task instructions. FST suggests a sharper criterion: information that is cheaply expressible in language and likely to be transient belongs in fast weights, while information that is diffuse, sub-symbolic, and broadly reusable belongs in slow weights. The interleaved training lets the system discover this allocation rather than imposing it by hand.

There is also a less obvious benefit. Because the fast context is a diverse population rather than a single prompt, the slow learner sees a richer conditioning distribution. I suspect part of the plasticity preservation comes not only from reduced KL but from this implicit regularization: the policy never gets to overfit to one prompt, so it never collapses its behavioral diversity as aggressively as pure RL does.

Limitations and Open Questions

Several caveats deserve honesty. First, the gains depend on tasks where rich textual feedback exists, math, code, and reasoning with verifiable signals and informative error traces. In domains where feedback genuinely is scalar (subjective preference, aesthetics), the fast channel has less to work with, and the advantage may shrink. Second, GEPA and reflective optimization add their own compute and their own LLM calls; the 3x rollout efficiency should be weighed against the cost of the fast loop, and the paper's accounting here matters. Third, the continual learning experiments use a small number of sequential tasks. Whether the plasticity benefit holds over long task sequences, where even small residual drift could accumulate, is an open empirical question.

A deeper question concerns scaling. As the context pool grows across many tasks, how is it managed? A Pareto frontier that keeps expanding eventually faces its own capacity and retrieval problems, which starts to look like a memory system rather than a prompt. The boundary between fast weights and external memory may turn out to be the next thing worth formalizing.

Conclusion

The contribution of Fast-Slow Training is conceptual as much as empirical. By splitting adaptation into a slow parametric channel that consumes scalar reward and a fast textual channel that consumes rich feedback, the authors show that LLM post-training does not have to mean compressing everything into weights. The payoff is a model that learns faster, drifts less, and, most importantly, stays able to learn again.

If the unifying claim holds, that forgetting and plasticity loss are the same problem of weights absorbing what context should hold, then the right design target for continual learning systems is not better regularization of weight updates but better routing of information between channels. That is a more optimistic picture than the usual continual learning story, because it suggests the bottleneck is architectural and addressable rather than fundamental.