Generalization and Trade-off in Adversarial Training: An RKHS Perspective via Kernel Integral Operator

This paper investigates the statistical properties of adversarial training within a nonparametric reproducing kernel Hilbert space (RKHS) framework. While adversarial training is widely used to improve model robustness, its impact on generalization performance—specifically the trade-off between robustness and statistical accuracy—remains poorly understood in nonparametric settings. The authors aim to characterize the approximation, estimation, and generalization errors of adversarial training estimators and determine if they can achieve minimax optimality.

The authors utilize the kernel integral operator to analyze the adversarial training estimator. By decomposing the generalization error into approximation and estimation components, they identify how the mixed robustness term (which arises from the adversarial objective) affects each. They discover that while this term provides beneficial curvature that reduces estimation complexity, it simultaneously introduces an irreducible noise component that slows the approximation rate. To mitigate this, they introduce a two-stage noise-debiased procedure that estimates the mean absolute noise and subtracts it from the objective function.

The study establishes that standard adversarial training in RKHS can be statistically suboptimal compared to the minimax prediction benchmark. Specifically, on models with fixed polynomial-spectrum decay, the generalization rate is slower than that of standard kernel ridge regression. The authors prove that their proposed noise-debiased estimator effectively removes the noise-induced approximation bottleneck, allowing the model to attain the minimax polynomial rate (up to logarithmic factors). Numerical experiments on synthetic and real-world datasets confirm that this debiasing approach improves generalization performance while maintaining robustness.