Transferable Physics-Informed Representations: Closing the Gap Between Learning and Inference in Scientific Machine Learning

Introduction

Physics-informed neural networks (PINNs) have emerged as a compelling bridge between deep learning and computational physics, offering a mesh-free alternative to traditional numerical methods for solving partial differential equations (PDEs). By embedding physical laws directly into the loss function, PINNs promise to learn solutions that respect governing equations, boundary conditions, and initial constraints without relying entirely on labeled data. Yet, despite their theoretical appeal, PINNs have struggled with practical deployment. Training these networks remains computationally expensive and numerically unstable due to stiff loss landscapes, and more critically, standard PINNs fail to generalize across problem instances. A model trained to solve the Navier-Stokes equations for one Reynolds number cannot readily adapt to another without extensive retraining.

In "Transferable Physics-Informed Representations via Closed-Form Head Adaptation", Wong et al. confront this limitation head on by reframing PINN training as a representation learning problem. Their framework, termed Pi-PINN (Pseudoinverse PINN), introduces a decoupled architecture that separates the learning of transferable physical features from task-specific solution mappings. The result is a system that can adapt to entirely new PDE instances, defined by new coefficients, source terms, or boundary conditions, through a closed-form pseudoinverse computation rather than iterative gradient descent. This approach achieves speedups of 100 to 1000 times compared to conventional PINNs while maintaining accuracy that exceeds purely data-driven models by one to two orders of magnitude in low-data regimes.

The Architecture of Closed-Form Adaptation

The core innovation of Pi-PINN lies in its architectural decoupling. Traditional PINNs map spatial and temporal coordinates directly to solution values through a monolithic deep network, optimizing all parameters simultaneously against a composite loss function involving PDE residuals, boundary conditions, and initial conditions. This entanglement means that every new PDE instance, parameterized by task-specific variables denoted as ϑ, requires re-optimizing the entire network.

Pi-PINN instead bifurcates the network into two components: a shared embedding network that learns a universal feature representation across related physical tasks, and a linear output head that maps these features to specific solution values. For a new PDE instance, the shared network generates embeddings for collocation points, and the optimal linear head is computed analytically via a least-squares pseudoinverse under PDE constraints. Mathematically, given the embedding matrix from the penultimate layer, the optimal weights w_L of the final layer solve a constrained linear system that minimizes the physics-informed loss without iterative backpropagation.

This closed-form adaptation exploits the linearity of the least-squares objective while respecting nonlinear physics constraints through careful construction of the pseudoinverse. The method effectively amortizes the cost of solving the PDE across multiple related instances during the meta-training phase, then enables rapid specialization during inference. Rather than transferring only initializations as in conventional fine-tuning approaches, Pi-PINN transfers a complete functional representation that requires only a linear solve to adapt.

Synergizing Physics and Multi-Task Learning

Beyond the architectural innovation, the paper provides crucial insights into the interplay between data-driven and physics-informed objectives. The authors employ a multi-task learning framework where the shared embedding is trained across diverse PDE instances from the same family, such as Poisson equations with varying source terms or Burgers equations with different viscosity coefficients. The training objective combines standard supervised losses on available solution data with physics-informed residuals across all tasks.

This hybrid approach addresses a fundamental tension in scientific machine learning. Purely physics-informed training often suffers from optimization pathologies, including spectral bias and unbalanced gradients between boundary and interior terms. Conversely, purely data-driven models lack generalization guarantees and require extensive labeled data. By learning a representation that must simultaneously satisfy physical constraints across multiple related tasks, the network discovers features that are inherently more physically consistent and numerically stable.

The empirical results demonstrate the potency of this synergy. On benchmark problems including the Helmholtz equation and nonlinear Burgers equation, Pi-PINN achieves relative errors 10 to 100 times lower than conventional data-driven models when provided with only two to four training samples per task. This data efficiency stems from the inductive bias imposed by the physics-informed multi-task objective, which effectively regularizes the representation space to contain only physically plausible solutions.

Broader Implications and Critical Analysis

Pi-PINN represents a significant step toward amortized inference in scientific computing, where the heavy computational lifting occurs during an offline training phase, enabling real-time adaptation during deployment. This paradigm shift is particularly relevant for applications requiring rapid parameter sweeps, inverse problem solving, or digital twins that must adapt to changing operational conditions in real time.

However, several limitations and assumptions warrant careful consideration. The framework assumes that PDE instances can be parameterized by a finite set of task parameters ϑ and that these instances share sufficient structure to permit transferable representations. For problems with discontinuous parameter regimes or qualitatively different physical behaviors, the shared embedding may fail to capture relevant features. Additionally, the restriction of the output head to a linear mapping, while enabling closed-form solutions, may limit expressiveness for highly nonlinear solution operators that cannot be factorized into a fixed nonlinear embedding followed by a linear projection.

The reliance on a pseudoinverse computation also introduces numerical considerations regarding the conditioning of the embedding matrix. For high-dimensional PDEs or problems requiring fine spatial discretization, the dimensionality of the linear system may grow prohibitively large, potentially offsetting the computational advantages of avoiding gradient-based optimization. The paper focuses primarily on two-dimensional problems; the scalability to three-dimensional turbulent flows or high-dimensional kinetic equations remains an open question.

From a broader perspective, Pi-PINN connects to the emerging literature on neural operators and meta-learning in scientific contexts. Unlike Fourier Neural Operators (FNOs) which learn mappings between function spaces in a fully data-driven manner, Pi-PINN explicitly incorporates differential operators into the adaptation mechanism. This suggests a promising middle path between the sample efficiency of physics-informed methods and the generalization capabilities of operator learning.

Conclusion

The work of Wong et al. demonstrates that the path toward practical, deployable PINNs requires rethinking not just the loss functions we optimize, but the very structure of the learning problem. By decoupling representation learning from task adaptation and exploiting the closed-form solvability of linear least-squares under physical constraints, Pi-PINN offers a template for building scientific machine learning systems that are both efficient and transferable.

Looking forward, several research directions appear fruitful. Extending the pseudoinverse framework to nonlinear heads through iterative closed-form updates could address expressiveness limitations while retaining computational advantages. Integrating Pi-PINN with adaptive meshing strategies or domain decomposition methods might enhance scalability to industrial-scale problems. Perhaps most intriguingly, the framework raises fundamental questions about the nature of physical similarity; what makes PDE instances "related" enough to share representations, and can we learn the appropriate metric of physical similarity itself?

As scientific machine learning matures, the distinction between training and inference continues to blur. Pi-PINN accelerates this convergence, suggesting a future where physical understanding is encoded not just in network losses, but in the very geometry of the learned representations.