Variational Boosting for Physics-Informed Neural Networks

Research Approach

Physics-Informed Neural Networks (PINNs) typically attempt to solve differential equations by training a single, large neural network to minimize a global residual functional. This approach often suffers from optimization instability, spectral bias, and ill-conditioning, particularly in multi-scale or nonlinear problems. The authors propose a variational boosting framework that replaces this monolithic approach with an additive, multi-stage construction. In this framework, the solution is built as a sequence of small 'weak learner' networks, where each stage trains a correction to the previous ensemble's output.

Key Mechanisms

The core innovation is the geometric interpretation of these stages as projected functional gradient descent. By restricting each stage to a small network, the authors transform the intractable global optimization problem into a series of well-conditioned subproblems. Because these individual correction networks are small, they allow for the application of second-order optimization methods—such as Newton or conjugate gradient updates—that are computationally infeasible for large, monolithic PINNs. The framework also incorporates a shrinkage parameter to control the contribution of each stage, ensuring stable convergence.

This method provides a theoretical bridge between PINNs and functional gradient boosting, offering a principled way to handle the evolving curvature of nonlinear differential operators. By separating the learning of different solution scales into sequential stages, the approach significantly improves training stability and performance. The ability to use second-order optimization at each stage allows for more precise solutions with fewer parameters and less training time compared to traditional, large-scale PINN architectures.