PAC–Bayes Bounds on Quotient Parameter Spaces: Geometry-induced Implicit-Bias Priors

This paper addresses the challenge of applying PAC-Bayesian analysis to overparameterized models, where multiple distinct parameter configurations can represent the same functional predictor. The authors propose moving the analysis from the raw parameter space to a quotient space, where all parameters corresponding to the same predictor are identified as a single point. By pushing the prior and posterior distributions to this quotient space, the authors eliminate the 'representational' component of the KL divergence—the portion of the complexity penalty that arises solely from how probability is distributed among equivalent parameters rather than from differences in prediction.

Beyond the quotient construction, the authors introduce a method to select a more informative prior. By analyzing the geometric volume of the symmetry orbits (the sets of equivalent parameters), they derive an 'implicit-bias prior.' This prior reflects the model's architecture and the natural tendencies of the optimization process, effectively approximating an ideal, data-dependent posterior-matched prior without violating the requirement that the prior must be independent of the certification data. The authors provide an exact criterion to determine when this geometric prior will yield a tighter generalization bound than a neutral reference prior.

To test their framework, the authors conducted experiments on sparse Fourier regression using a Hadamard parameterization and on Query-Key attention mechanisms. In the Fourier-Hadamard setting, using standard SGD without explicit regularization, the implicit-bias prior reduced the mean quotient-space KL divergence by 40.69% and the overall PAC-Bayes certificate by 21.40%. The results in the Query-Key attention experiment were more modest and scale-dependent, confirming that the benefits of the geometric prior are conditional on the alignment between the prior and the learned posterior.