An Optimisation Framework for the Well-Conditioned Training of Physics-Informed Neural Networks

Physics-Informed Neural Networks (PINNs) are powerful tools for solving partial differential equations (PDEs), but they often struggle to match the precision of classical numerical solvers. This paper investigates the hypothesis that the primary bottleneck is the ill-conditioned loss landscape, which standard first-order optimizers fail to navigate effectively.

The authors propose DSGNAR (Doubly-Sketched Gauss-Newton with Adaptive Ratio), a second-order optimization framework. The method employs two key innovations:

1. Doubly-Sketched Gauss-Newton: Instead of storing the full, computationally expensive Jacobian, the authors use CountSketch to compress the residual dimension and a Subsampled Randomized Cosine Transform (SRCT) to compress the parameter dimension. This allows for an explicit, memory-efficient construction of a small, square sketched Jacobian.
2. Adaptive Ratio Step Selection: Rather than manually tuning regularization parameters or trust-region radii, the algorithm probes multiple candidate steps and selects the one that achieves a target decrease ratio (the ratio of actual objective reduction to predicted reduction). This ratio is dynamically adjusted to first prioritize conditioning the problem and subsequently to accelerate descent.

DSGNAR demonstrates significant performance gains across a diverse suite of PDEs, including nonlinear, chaotic, multi-scale, and high-dimensional problems. Key results include:

• Unprecedented Accuracy: The framework achieves relative L2 errors as low as 3x10^-16 in double precision, outperforming existing benchmarks by up to eight orders of magnitude in high-dimensional Poisson problems.
• Efficiency: The method remains highly performant in single precision, solving the Burgers' equation to 4.75x10^-7 error in under ten seconds.
• Robustness: The framework is largely independent of specific network architectures (e.g., MLPs, SIRENs, GaborNets) and initial hyperparameters, making it a versatile tool for scientific machine learning.