Cognitive Primitives in Modular Language Models: What Brainstacks Reveals About Fine-Tuning

When a medical question about drug interactions routes through chat and mathematics adapters to reach an optimal answer, we must reconsider what fine-tuning actually accomplishes. In Brainstacks: Cross-Domain Cognitive Capabilities via Frozen MoE-LoRA Stacks for Continual LLM Learning, Mohammad R. Abu Ayyach presents an architecture that yields a striking empirical discovery: domain specific fine-tuning appears to install transferable cognitive operations rather than domain specific knowledge repositories. The system achieves 97% optimal routing of medical prompts to stacks trained exclusively on conversational data and mathematical reasoning, despite those stacks containing zero medical training examples. This result challenges the prevailing metaphor of fine-tuning as knowledge retrieval, suggesting instead that we are observing the emergence of composable cognitive primitives.

The Architecture of Frozen Composability

Brainstacks operates through five interlocking mechanisms that together enable continual learning without catastrophic forgetting. At its core lies a Mixture-of-Experts LoRA (MoE-LoRA) block applied to all seven transformer projection matrices, including the query, key, value, and output projections of attention layers. Unlike prior MoE-LoRA implementations that restrict routing to feedforward layers or use simplified gating, this architecture employs Shazeer-style noisy top-2 routing with learnable noise injection via a dedicated noise linear layer using softplus activation. Combined with rsLoRA rank stabilized scaling and QLoRA 4-bit quantization, this configuration demonstrates 2.5× faster convergence than parameter matched single LoRA baselines on both TinyLlama-1.1B and Gemma 3 12B IT models.

The training procedure bifurcates into an inner loop and outer loop that together implement residual boosting. The inner loop freezes trained MoE-LoRA stacks and adds new residual stacks that learn what previous stacks failed to capture, effectively performing gradient boosting in the function space of low rank adaptations. The outer loop sequences domain specific training with curriculum ordered dependencies, allowing the system to accumulate nine to ten distinct domain stacks without interference. Crucially, null-space projection via randomized SVD constrains new stack gradients to subspaces orthogonal to previously claimed directions, providing architectural guarantees of zero forgetting when domains are evaluated in isolation.

At inference, an outcome-based sigmoid meta-router selectively weights stacks based on empirically discovered domain combination targets rather than predefined labels. This router enables the Superposition LLM principle: disk offloaded selective loading allows arbitrary numbers of domain stacks with constant GPU memory, as only relevant stacks are materialized for each prompt.

From Knowledge Retrieval to Cognitive Operations

The central empirical contribution of Brainstacks reframes the ontology of fine-tuning. The outcome-based router discovers that domain stacks encode transferable primitives: instruction following clarity, numerical reasoning, procedural logic, and chain-of-thought structure. When processing medical prompts, the router assigns weight to chat stacks for linguistic clarity and mathematical stacks for quantitative reasoning, despite neither stack encountering medical data during training.

This finding suggests that the distinction between domain knowledge and cognitive capability, often treated as separate in AI discourse, may be largely artificial at the level of neural representations. The medical domain appears to decompose into underlying cognitive operations that exist independently across training distributions. Rather than retrieving medical facts from a specialized database, the model composes general reasoning primitives that happen to suffice for medical question answering in this experimental setup.

The implication extends beyond technical architecture into epistemology. If fine-tuning primarily installs cognitive operations rather than domain specific knowledge, then the traditional view of LLMs as retrieval systems with parametric memory requires revision. The Brainstacks results suggest that continual learning systems may be better understood as accumulating reusable reasoning strategies, with domain specificity emerging from the composition of these primitives rather than from isolated knowledge silos.

Limitations and Broader Context

Despite the compelling results, several constraints warrant careful consideration. The 97% routing success for medical prompts demonstrates strong correlation between medical reasoning and the chat plus math primitive combination, yet this may fail for medical queries requiring specialized ontologies or rare disease knowledge that genuinely requires domain specific parametric encoding. The null-space projection guarantees zero forgetting for isolated domains, but the paper does not fully characterize interference patterns when multiple stacks are simultaneously active through the meta-router, particularly for out of distribution compositional queries.

Furthermore, the reliance on QLoRA 4-bit quantization, while enabling practical deployment, may introduce precision limitations for numerical reasoning tasks requiring high fidelity arithmetic. The randomized SVD projection, though theoretically elegant, adds computational overhead during training that may scale poorly as the number of accumulated stacks grows into the hundreds.

Compared to prior continual learning approaches such as Elastic Weight Consolidation or PackNet, Brainstacks offers genuine compositionality at inference, not merely protection against forgetting. Unlike LoRAMoE, which addresses knowledge forgetting during instruction tuning through localized balancing constraints, Brainstacks permits arbitrary addition and removal of capabilities post-deployment. The residual boosting mechanism resembles progressive neural networks in its lateral connections, yet differs fundamentally by sharing a frozen base model and utilizing routed expert ensembles rather than standard single adapters.

Original Insights: The Substrate First Hypothesis

The Brainstacks findings support what might be termed the Substrate First Hypothesis: effective domain mastery in neural systems depends more on developing robust general reasoning substrates than on accumulating domain specific parameters. This aligns with observations in human education, where mathematical reasoning and logical clarity transfer across disciplinary boundaries, yet carries risks if interpreted too strongly. While cognitive primitives may suffice for structured medical question answering in benchmark settings, specialized domains likely require both general reasoning capabilities and specific factual grounding.

The meta-router's empirical discovery of optimal compositions suggests a path forward for automated architecture search. Rather than hand designing task specific model variants, future systems might maintain libraries of frozen cognitive primitives and learn to route among them based on outcome optimization. This raises profound questions about the limits of such composition. How many distinct cognitive primitives exist? Are they universal across model scales, or do larger models develop more granular and transferable operations? Can we discover these primitives systematically without exhaustive experimental routing?

The disk offloaded inference system points toward a future where organizations maintain private stacks of proprietary cognitive capabilities, composing them on demand atop shared foundation models. This modular approach addresses not only continual learning but also intellectual property concerns, as individual primitives can be licensed, updated, or revoked independently.

Conclusion

Brainstacks demonstrates that the future of continual learning lies not in protecting monolithic knowledge bases from forgetting, but in architecting systems that compose frozen cognitive primitives. The finding that medical reasoning decomposes into conversational clarity and mathematical rigor suggests that the space of human cognitive capabilities may be smaller and more structured than the space of human knowledge domains.

Looking forward, the field must investigate whether these primitives are discovered or created during training, whether they converge across different base model architectures, and how they interact when composed in novel combinations never seen during meta-router training. The transition from viewing fine-tuning as knowledge injection to understanding it as cognitive operation installation represents a necessary conceptual evolution as we build increasingly modular and composable AI systems. The question is no longer how much a model knows, but which reasoning substrates it possesses and how effectively they can be orchestrated.