Covariance Last-Layer Ensembles: Function-Space Diversity for Efficient Uncertainty Quantification

Last-layer ensembles (LLEs) are computationally efficient methods for estimating epistemic uncertainty by training multiple linear heads on a shared, frozen feature map. However, because these heads share the same backbone gradients, they often converge to identical functions, leading to a collapse in inter-member diversity. This paper investigates whether this collapse can be mitigated in function space to recover the performance of full deep ensembles without the associated training and inference costs.

The authors propose the Covariance Last-Layer Ensemble (cov-LLE), which introduces a covariance penalty on member activations during training. This approach is contrasted with Orthonormal Certificates (OC), which attempt to decorrelate members in weight space. The authors also organize existing OOD detectors into a 2x2 taxonomy based on their training objective (unsupervised vs. supervised) and scoring rule (norm vs. disagreement), identifying that label-free failures in OC are primarily scoring artifacts that can be corrected with a scale-invariant direction score.

• Function-Space Diversity: The cov-LLE successfully restores function-space diversity that weight-orthonormality cannot, achieving prediction variance and calibration metrics that approach those of a full deep ensemble at only 1x backbone cost.
• Performance Gains: On standard benchmarks, cov-LLE significantly outperforms weight-orthonormal LLEs on far-OOD detection tasks without sacrificing classification accuracy.
• Taxonomy and Scoring: The authors demonstrate that the label-free failure of OC methods is a scoring artifact. By implementing a scale-invariant direction score, they improve ROC AUC by +0.16 to +0.18 across various backbones, effectively repairing near-OOD detection performance.