Separable Neural Architectures: Unifying Prediction and Generation Through Factorised Representations

The dominance of monolithic neural architectures, from convolutional networks to Transformers, has defined the trajectory of modern artificial intelligence. These models treat the systems they approximate as undifferentiated black boxes, learning high dimensional mappings through sheer parametric scale. Yet physical, linguistic, and perceptual systems frequently exhibit latent factorisable structure that these architectures leave implicit rather than exploit. In their recent work, "Separable neural architectures as a primitive for unified predictive and generative intelligence," Batley et al. propose a compelling alternative: the Separable Neural Architecture (SNA). This formalism treats separability not merely as a mathematical convenience, but as a fundamental representational primitive capable of unifying predictive and generative modelling across disparate domains. By constraining interaction order and tensor rank, SNAs impose a structural inductive bias that factorises high dimensional mappings into low arity components, offering a parsimonious alternative to the prevailing trend of unrestricted parametric expansion.

The Mechanics of Separability

At the core of the SNA formalism lies the interaction object, a sparse tensor that governs how low arity learnable components, termed atoms, combine to construct global representations. Expressivity is controlled through two primary hyperparameters: the tensor rank $r$ and the interaction order $k$. By adjusting these controls, the architecture subsumes additive models, quadratic interactions, and general tensor decomposed neural networks within a single framework. In the specific CP class (Canonical Polyadic) instantiation, which permits full interaction ($k=d$) and factorises the interaction tensor into rank $r$ components, atoms decompose into products of univariate sub atoms $\psi$. The resulting mapping takes the form:

$$f_r(x; \Theta_{CP}) = \rho\left(\sum_{j=1}^{r} c^{(j)} \prod_{i=1}^{d} \psi^{(j)}(x_i; \theta^{(k)}_d)\right)$$

Here, $c^{(j)}$ represents modal weights and $\rho$ denotes an activation function. This formulation is instantiated concretely in KHRONOS, an SNA employing identity activation, unit modal weights, and B spline sub atoms. Unlike standard multilayer perceptrons, which rely on compositional non linearities across layers, KHRONOS operates as a smooth, cheaply invertible interpolant over the input space. Despite utilizing only hundreds of trainable parameters, it achieves accurate prediction and rapid generative inversion to recover entire manifolds of admissible inputs. This efficiency stands in marked contrast to the millions or billions of parameters typical in contemporary foundation models, suggesting that explicit structural constraints can yield significant computational advantages when the underlying physics or logic permits factorisation.

A Structural Analogy Across Domains

Perhaps the most intellectually provocative contribution of this work is the identification of a structural analogy between chaotic spatiotemporal dynamics and linguistic autoregression. The authors observe that chaotic systems preclude stable pointwise prediction over extended horizons; modelling must therefore be distributional to avoid nonphysical drift. Under this view, predicting the next state of a turbulent fluid and predicting the next token in a sentence become formally similar tasks, both requiring the modelling of conditional distributions over sequentially revealed states. This insight motivates the treatment of continuous physical states as smooth, separable embeddings, enabling distributional modelling of chaotic systems while remaining applicable to discrete sequences.

This coordinate aware formulation reveals that separability need not be an intrinsic property of the system itself, but often emerges in the coordinates or representations through which the system is expressed. The authors demonstrate this versatility across four distinct domains: autonomous waypoint navigation via reinforcement learning (SPAN), inverse generation of multifunctional microstructures (Janus), distributional modelling of turbulent flow, and neural language modelling (Leviathan). In the physical contexts, the Variational Separable Neural Architecture (VSNA) reinterpretes the SNA as a structured Galerkin trial space, learning high dimensional spatiotemporal fields directly from governing operators. For language modelling, the approach enables continuous token embeddings that preserve neighbourhood relations in the underlying state space, a feature absent from the discrete lookup embeddings prevalent in current sequence models. This domain agnostic applicability positions the SNA as a genuine primitive, capable of serving as a standalone model, a variational trial space, or a compositional module within larger intelligent systems.

Limitations and the Cost of Structure

Despite these compelling results, the SNA framework carries inherent limitations that merit careful consideration. The primary constraint lies in the assumption of separability itself. While the authors correctly note that suitable coordinate systems can induce separability, there is no guarantee that such coordinates exist or can be discovered efficiently for arbitrary high dimensional systems. Complex, highly coupled phenomena may resist factorisation into low rank, low order components, rendering the SNA less expressive than monolithic alternatives for certain pathological cases. Furthermore, the choice of rank $r$ and interaction order $k$ introduces a model selection problem that requires domain expertise or expensive validation, potentially offsetting the parameter efficiency gains in data scarce regimes.

The reliance on B spline sub atoms in the KHRONOS instantiation, while providing smoothness and invertibility, may also limit the model's ability to capture sharp discontinuities or singularities present in certain physical systems. Additionally, the optimization landscape for tensor decomposed architectures can exhibit spurious local minima or degeneracies when modes collapse, requiring careful initialization and regularization strategies that the paper does not fully address. While the authors demonstrate success on specific benchmarks, the scalability of these methods to web scale language models or extremely high dimensional chaotic systems (beyond the turbulence examples provided) remains an open empirical question. The tension between the interpretability of separable structures and the raw expressivity of unrestricted attention mechanisms suggests that SNAs may find their greatest utility as specialized modules rather than universal replacements for existing architectures.

Reinterpreting the Role of Structure in Learning

Beyond the technical specifications, this work represents a necessary correction in how we conceptualize the relationship between architecture and problem structure. The prevailing consensus in machine learning has favored domain agnostic monoliths, assuming that sufficient data and compute can overcome structural mismatches between model and target. The SNA framework argues persuasively for the opposite: that intelligence, whether artificial or natural, operates most efficiently when representational primitives align with the factorisable structure of the world. The coordinate aware perspective is particularly significant; it shifts the burden of separability from the system to the representation, implying that much of learning consists of discovering the right coordinate system in which to be separable.

This insight connects SNAs to broader movements in physics informed machine learning and disentangled representation learning. By unifying predictive and generative capabilities within a single lightweight architecture, KHRONOS challenges the specialization that typically separates discriminative and generative modelling into distinct architectural families. The ability to invert predictions efficiently, recovering entire manifolds of solutions consistent with an output constraint, suggests applications in scientific inverse problems where traditional simulators require expensive Markov chain Monte Carlo sampling. However, I suspect the most impactful application of SNAs will be compositional; as the authors demonstrate with SPAN and Janus, these primitives can serve as structured inductive biases within larger hybrid systems, handling the separable components of a problem while monolithic architectures manage irreducible complexity.

Conclusion

The separable neural architecture offers a rigorous mathematical framework for exploiting latent factorisable structure across intelligent systems. By formalising rank and interaction controlled mappings, Batley et al. provide a tool that unifies predictive and generative modelling from turbulent flows to linguistic sequences. Looking forward, several questions remain ripe for investigation. Can we develop meta learning algorithms that automatically discover separable coordinate systems for arbitrary data, or must this remain a manual art? How do approximation theoretic bounds for the CP class compare to universal approximation guarantees for standard neural networks under realistic smoothness constraints? Finally, can SNAs be scaled to compete with foundation models on massive text corpora without sacrificing their invertibility and interpretability?

As the field grapples with the computational and environmental costs of monolithic scale, the SNA represents a compelling alternative path. It suggests that intelligence need not emerge from undifferentiated mass, but can arise from the careful composition of structured, separable parts. Whether this primitive supplants or merely complements current approaches will depend on our ability to identify and exploit the hidden factorisations present in the world's most complex phenomena.