Checkpoints as Experts: Rethinking Dense-to-MoE Upcycling

The Mixture-of-Experts (MoE) architecture has become one of the more practical answers to a stubborn problem in large language models: how to grow model capacity without paying a proportional cost at inference time. By activating only a subset of parameters per token, MoE models decouple total capacity from per-token compute. The catch is that training a good MoE model from scratch is expensive, which has motivated a body of work on upcycling, the practice of converting an existing dense model into an MoE one.

A recent paper, "Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging" (UpIT) by Hui et al., proposes a notably data-efficient route through this conversion. The central observation is simple enough to be easy to overlook: intermediate checkpoints saved during ordinary instruction tuning already behave like specialized experts. This post examines how UpIT works, why the observation is interesting, and where I think the approach has both real strengths and unresolved tensions.

The Two Existing Upcycling Recipes and Their Costs

Before UpIT, dense-to-MoE upcycling came in two flavors, both data-hungry.

Vanilla upcycling, introduced in work like Komatsuzaki et al.'s sparse upcycling, replicates the feed-forward network (FFN) layers of a dense model to create multiple experts, attaches a router, and then runs large-scale post-training to differentiate the experts. Because all experts start as exact copies, the initialization carries no diversity at all. The router has nothing meaningful to choose between, so the model needs enormous amounts of post-training to break the symmetry. The paper cites figures on the order of 1T tokens for full-parameter training, or roughly 5M instruction examples for parameter-efficient variants.

Specialized upcycling, as in Sukhbaatar et al.'s Branch-Train-MiX, instead trains distinct experts on carefully curated domain-specific datasets first, then merges them into an MoE and post-trains. This produces genuinely differentiated experts and better performance, but it requires hundreds of billions of tokens of elaborately constructed domain data, and it is inflexible: the number of experts is fixed by the number of domain datasets you bothered to build.

The shared problem is clear. Both approaches treat the construction of diverse experts as something you must pay for, either with post-training compute or with curated data engineering.

The UpIT Insight: Checkpoints Are Already Specialized

UpIT's pilot experiment is the heart of the paper. The authors take a dense model through standard instruction tuning and evaluate checkpoints saved at different epochs across nine benchmarks, grouped into four domains: factual knowledge (ARC, MMLU), reasoning (HellaSwag, GSM8K, BBH), coding (HumanEval), and world knowledge (TriviaQA, Natural Questions).

The result is that different checkpoints peak on different benchmarks. The model at epoch 2 leads on HellaSwag and Natural Questions, while the checkpoint at epoch 0.25 is best on MMLU. Each benchmark generally rises then falls over training, but the location of the peak varies by domain. In other words, the optimization trajectory naturally passes through regions of parameter space that favor different competencies. Rather than discarding all but the final checkpoint, UpIT proposes harvesting these intermediate states as ready-made specialized experts.

From this observation, the method proceeds in four stages:

• Expert Preparation. Fine-tune the dense model and save checkpoints at fixed intervals. No careful selection is needed; the trajectory provides natural diversity.
• Expert Expansion. Because the number of useful checkpoints may not match the desired number of experts, the authors expand the pool using a genetic algorithm. At each iteration they select the two most dissimilar experts, merge their parameters, and produce a new expert. They additionally apply parameter scaling and dropping before merging, a deliberate analogue of mutation, to push the experts further apart.
• Router Initialization. Instead of randomly initializing the router, which would be insensitive to the specialization that already exists, they select a small amount of seed data that each expert excels at and use it to pre-optimize the router so it routes inputs to the appropriate expert.
• Post-Training. A final, comparatively lightweight tuning stage.

The payoff is data efficiency. Rather than 1T tokens or millions of curated examples, UpIT leans on the structure already present in a single instruction-tuning run plus a small seed set, and reports stable improvements as either experts or data are scaled.

Why the Genetic Merging and Router Steps Matter

Two design choices deserve emphasis because they reveal what is really doing the work.

The genetic algorithm with maximum-difference selection is an explicit acknowledgment that the raw checkpoints are not diverse enough on their own. If you simply replicate a checkpoint, you reproduce the vanilla-upcycling problem. By repeatedly merging the two most dissimilar experts and injecting mutation through scaling and dropping, the method tries to manufacture diversity rather than assume it. The authors' own ablations reportedly confirm that ensuring expert diversity is the key factor in upcycling performance, which suggests the merging step is not a convenience but a load-bearing component.

The router pre-optimization on skilled seed data is the second crucial piece. A standard MoE trains the router jointly from random initialization, which means early in training the router routes tokens essentially at random. UpIT instead grounds the router in evidence about which expert is genuinely better at which kind of input. This converts router learning from a cold-start problem into a warm-start one, which is precisely why so much less post-training data is needed.

My Take: Is Trajectory Diversity the Same as Functional Specialization?

Here is where I want to push beyond the paper's framing. The approach is elegant, and the data-efficiency gains are real, but the conceptual foundation has a subtle weakness.

All of UpIT's base experts come from a single training run with a shared ancestor. The diversity among checkpoints is diversity along one optimization trajectory. The model passes through different loss-landscape regions, and these regions happen to favor different benchmarks at different times. But this is not the same as the functional specialization you get when you train separate models on genuinely different data distributions, as specialized upcycling does.

The risk is that we are rediscovering correlated minima and labeling them experts. Two checkpoints separated by half an epoch may differ in their benchmark scores, but those differences could reflect transient overfitting and forgetting dynamics rather than stable, complementary skills. The benchmark peaks the paper reports might partly be the model trading one competency for another as it moves, which is a redistribution of a fixed budget rather than the creation of orthogonal expertise. If that is the case, the upper bound on expert diversity is set by the optimization trajectory itself, not by the genuine breadth of skills the model could host.

This is why I suspect the router pre-optimization is doing more of the heavy lifting than the expert count. If the experts are only weakly differentiated, then the seed data that the router is optimized on becomes the true determinant of performance, because it is what defines the boundaries between overlapping competencies. The real bottleneck is then not how many experts you generate through genetic merging, but whether the seed set actually disentangles competencies that overlap substantially in parameter space. A poorly chosen seed set would leave the router unable to separate experts whose behaviors are nearly collinear.

A clean test would be to measure the functional, not just benchmark-level, divergence between experts: per-token routing entropy after training, the degree of expert collapse, and whether ablating individual experts produces domain-localized degradation. If removing the "coding" expert specifically harms HumanEval and little else, the specialization is real. If it harms everything uniformly, the experts are interchangeable and the MoE structure is decorative.

Conclusion

UpIT makes a genuinely useful contribution: it shows that the artifacts most instruction-tuning pipelines already throw away, the intermediate checkpoints, carry exploitable specialization, and it combines genetic parameter merging with router pre-optimization to turn them into a working MoE with a fraction of the usual data. The data-efficiency story is convincing, and the emphasis on expert diversity as the controlling variable is the right diagnosis.

The open question I keep returning to is whether trajectory-derived diversity can ever match data-derived specialization, or whether it is fundamentally bounded by how far a single run can wander. The most interesting follow-up work would combine multiple short, divergent fine-tuning runs as the checkpoint source, giving the genetic merging step genuinely distinct ancestors to recombine. If that closes the gap with specialized upcycling at comparable cost, the case for UpIT becomes much stronger. Until then, the method is a clever and efficient construction whose experts may be more correlated than the word "expert" suggests.