Efficient Bayesian Deep Ensembles via Analytic Predictive Inference

Predictive regression models in safety-critical applications require both high predictive accuracy and reliable uncertainty quantification. While deep ensembles are scalable and performant, they lack a formal probabilistic foundation, often leading to overconfident predictions. Conversely, fully Bayesian methods provide rigorous uncertainty but are computationally prohibitive. This paper addresses the challenge of creating a model that balances computational efficiency, strong performance, and interpretable uncertainty.

The authors propose Bayesian Deep Kernel Networks (BDKN), which combine the scalability of deep ensembles with the statistical rigor of Bayesian inference. The framework consists of three stages:

1. Independent Ensemble Training: Multiple neural networks are trained separately to produce diverse feature representations.
2. Feature Space Representation: These networks act as basis functions, mapping inputs into a low-dimensional feature space.
3. Closed-Form Bayesian Aggregation: Instead of uniform averaging, the model applies Bayesian linear regression on top of the ensemble outputs. By placing a conjugate prior on the linear weights and the noise variance, the model derives an analytic posterior predictive distribution (a Student-t distribution) without requiring expensive approximate inference like variational methods.

BDKN consistently achieves competitive predictive performance (RMSE) and superior uncertainty calibration (NLL) compared to standard deep ensembles and various Bayesian baselines across standard UCI regression benchmarks. Because the Bayesian aggregation is performed in a low-dimensional space defined by the ensemble size, the additional computational cost is negligible—scaling linearly with the dataset size rather than cubically like traditional Gaussian Processes. The framework also demonstrates increased robustness to optimization variability, as the Bayesian posterior automatically downweights ensemble members that are poorly supported by the data.