PRISM: Sensitivity-Aware PolynoMial PRuning for EffIcient Neural Network Encryption

As neural networks are increasingly deployed in privacy-sensitive domains using Homomorphic Encryption (HE), researchers face a dual challenge: the extreme computational cost of HE operations and the vulnerability of these models to transient hardware faults (e.g., bit-flips in memory). Existing pruning methods focus on computational efficiency but ignore how these optimizations affect the reliability of the underlying encrypted computation. This paper introduces Polynomial-Sensitivity-Aware Pruning (PSAP), a framework that optimizes neural networks for the CKKS encryption scheme by jointly considering weight magnitude, polynomial activation sensitivity, and rotation costs.

The PSAP framework operates in four stages: converting standard ReLU activations into trainable polynomial approximations, characterizing the sensitivity of these activations, applying a reliability-aware pruning strategy, and performing quantization-aware fine-tuning. By calculating an activation sensitivity score—based on the gradient of the polynomial activation over the input distribution—the framework identifies which filters are most critical for information propagation. It then prioritizes keeping these filters while pruning those that are less sensitive and more computationally expensive to rotate in the encrypted domain.

PSAP demonstrates a significant improvement in model robustness. In experiments across various datasets and architectures, PSAP-pruned models limited catastrophic accuracy drops (defined as >10 percentage points) to at most two layers, compared to 5–14 layers in standard magnitude-pruned baselines. This represents a reduction in worst-case vulnerability by up to 29 times under bit-flip injection. Beyond reliability, the method achieves substantial efficiency gains, reducing Halevi–Shoup rotations by up to 45.2% and lowering the multiplicative depth of the networks, which allows for leveled inference without the need for costly bootstrapping.