The Hidden Curriculum: How Language Models Learn in Predictable Sequences

Introduction

We have grown accustomed to watching loss curves descend during pretraining, treating the smooth exponential decay of perplexity as a proxy for learning. Yet this aggregate metric conceals as much as it reveals. A model's cross-entropy loss on held out data decreases predictably with scale, following the power laws that have become gospel in the field. But when does a model learn to track coreference? At what point does arithmetic ability crystallize? And perhaps more importantly, do these capabilities emerge in random order, or does pretraining follow a structured trajectory that is consistent across architectures and data mixtures?

In their recent preprint, "What Do Language Models Learn and When? The Implicit Curriculum Hypothesis," Emmy Liu and colleagues propose a compelling answer: language models follow an implicit curriculum during pretraining, acquiring skills in a compositional and predictable order that is readable from their internal representations. Their findings suggest that the black box of pretraining is less opaque than we assumed, and that our reliance on loss curves as primary diagnostics may be fundamentally misguided.

The Architecture of Emergence

The researchers designed a suite of elemental tasks probing specific capabilities: retrieval, morphological transformations, coreference resolution, logical reasoning, and mathematical operations. These were carefully constructed to be composable; for instance, a task requiring French to English translation followed by uppercasing represents a two step composition of elemental skills. By tracking performance across nine models from four families (Pythia, OLMo 2, Amber, and others) ranging from 410M to 13B parameters, they mapped the emergence points of these capabilities throughout training.

The consistency they observed is striking. Across 45 model pairs, the Spearman correlation between emergence orderings reached ρ = .81. Whether training Pythia or OLMo, whether at 410M or 7B parameters, the sequence holds: copying emerges first, followed by simple string manipulations and morphological transformations, then fact extraction and coreference, then logical operations, and finally multi step arithmetic and complex reasoning. This is not merely a story of "simple before complex." The researchers found that composite tasks consistently emerge after their constituent elemental tasks, suggesting that the model is not memorizing end to end solutions but rather composing previously acquired primitives.

A crucial technical nuance emerges in their methodology. This consistency only holds when emergence is defined by fixed accuracy thresholds (specifically, the 50% mark), not relative thresholds. When measured relative to each model's final performance, the ordering appears more variable. This distinction matters for experimental design; it suggests that the implicit curriculum has absolute milestones rather than merely relative progressions, and that comparing models at fixed thresholds reveals structural similarities that relative metrics obscure.

Representational Prognosis

Perhaps the most significant contribution lies in the paper's analysis of internal representations. Using function vectors derived from the residual stream (following Todd et al., 2024), the researchers measured the geometric proximity of task representations within the model's activation space. They discovered that tasks with similar function vectors follow similar learning trajectories. This correlation between representational geometry and learning dynamics is profound; it suggests that the model's internal organization foreshadows its developmental trajectory.

Leveraging this insight, the authors demonstrate that they can predict the full training trajectory of held out compositional tasks without ever evaluating them during training. By situating the new task in the representational space derived from their elemental task suite, they achieve R² values of .68 to .84 across models in predicting when capabilities will emerge, with per task R² exceeding .95 for certain benchmarks. This representational predictability transforms interpretability from a post hoc analysis tool into a predictive instrument for training diagnostics.

Rethinking Training Methodology

These findings carry substantial implications for how we approach large scale pretraining. Currently, million dollar training runs are monitored primarily through aggregate loss and sporadic downstream benchmark evaluations. When performance on GSM8k stalls, practitioners face a diagnostic dead end; the benchmark requires numerical fluency, multi step planning, and natural language understanding in inseparable combination. The Implicit Curriculum Hypothesis suggests an alternative: by probing elemental skills and examining internal representations, we can identify specific bottlenecks in the learning process.

The work also reframes the discourse around "emergence" in large language models. Rather than viewing abrupt capability gains as mysterious phase transitions or mere artifacts of evaluation metrics, this research suggests they are predictable consequences of compositional skill acquisition. The curriculum is implicit, not explicit; no human designer programmed this sequence. Yet the structure emerges inevitably from the statistical structure of language and the optimization dynamics of transformers, much as developmental stages emerge in human cognition.

There are, however, limitations that warrant honest consideration. The study's scale ceiling of 13B parameters, while respectable, leaves open questions about whether these patterns hold for the 100B+ models that dominate current capabilities research. The task suite, though carefully designed, remains synthetic; it probes explicit reasoning chains more readily than the implicit, associative knowledge that characterizes much of human language use. Furthermore, while the correlation between representational proximity and learning trajectories is robust, the causal mechanisms remain unclear. Do tasks cluster in representation space because they share computational primitives, or does the geometry of the loss landscape drive both representational similarity and learning order?

Open Questions and Forward Trajectories

The most tantalizing implication of this work concerns the potential for active curriculum design. If pretraining naturally follows a compositional curriculum, could we accelerate learning by structuring data ordering to match this implicit sequence? Or conversely, could we disrupt harmful capabilities by understanding and avoiding their developmental prerequisites? The function vector framework suggests that we might eventually steer training trajectories not just through data curation, but through direct manipulation of representational geometry.

Looking ahead, several research directions appear promising. Extending this analysis to multimodal models could reveal whether vision language models follow similar compositional hierarchies, or whether cross modal capabilities emerge through distinct pathways. Investigating the relationship between the implicit curriculum and mechanistic interpretability frameworks like circuit tracing could bridge the gap between macroscopic developmental patterns and microscopic algorithmic implementations. Finally, understanding why the curriculum takes its specific shape, connecting the statistical structure of pretraining data to the observed emergence order, remains the fundamental theoretical challenge.

The Implicit Curriculum Hypothesis invites us to view pretraining not as an undifferentiated process of pattern absorption, but as a structured developmental trajectory with predictable milestones. As we train ever larger models, the ability to read their internal clocks, to know what they have learned and what they are ready to learn next, may prove more valuable than any aggregate metric. The loss curve tells us that learning is happening; this research begins to tell us what that learning actually consists of.