Physics-Informed Neural Network with Squeeze-Excitation-like Attention

Overview

Physics-Informed Neural Networks (PINNs) are powerful tools for solving partial differential equations (PDEs), yet they often suffer from high sensitivity to initialization, slow convergence, and difficulty capturing high-frequency features. This paper introduces SEA-PINN, a novel architecture that incorporates a Squeeze-Excitation-like attention mechanism to dynamically recalibrate the importance of neurons across layers. By doing so, the model achieves a more stable and favorable starting point for optimization.

The authors adapt the Squeeze-and-Excitation (SE) block—originally designed for convolutional neural networks—to the neuron level within a standard fully connected PINN. In this architecture, a lightweight weight generator computes neuron-wise attention weights based on the layer's output. These weights are then applied via element-wise multiplication to rescale activations before they propagate to the next layer. This mechanism allows the network to adaptively adjust its internal representation during training without requiring complex modifications like Fourier feature embeddings or periodic activation functions.

Key Findings

SEA-PINN demonstrates superior initialization characteristics compared to standard FNN-PINNs and even specialized architectures like TSA-PINN. Across 20 benchmark problems, SEA-PINN consistently exhibits lower initial loss and significantly reduced variance across multiple random seeds. The authors show that this stability stems from the network's ability to inherently satisfy PDE and boundary condition constraints at initialization. Furthermore, Jacobian-based analysis reveals that SEA-PINN maintains smoother input-output mappings and better-conditioned information propagation between layers, which facilitates more robust and efficient convergence.

Training instability is a primary bottleneck for the widespread adoption of PINNs in scientific computing. SEA-PINN offers a lightweight, plug-in solution that enhances the reliability of these models. By providing a quasi-deterministic starting point, it reduces the need for extensive hyperparameter tuning and multiple trial runs, making physics-informed learning more efficient and accessible for complex, high-dimensional problems.