RETHINKING LIKELIHOOD DISTRIBUTIONS: STUDENT’S T LIKELIHOOD BOOSTS BAYESIAN NEURAL NETWORK PERFORMANCE

Bayesian neural networks (BNNs) are essential for quantifying uncertainty in machine learning, yet they often rely on the assumption that the likelihood distribution is Gaussian. This choice is primarily driven by mathematical convenience—specifically, the ease of computing gradients and closed-form loss functions—rather than empirical evidence. The authors investigate whether this standard Gaussian assumption leads to model misspecification and if alternative likelihood distributions, such as the heavy-tailed Student's t or skewed distributions, can provide superior predictive performance.

The researchers conducted a systematic evaluation of different likelihood distributions within the variational inference (VI) framework. They compared the standard Gaussian likelihood against alternatives like the Student's t and skew-normal distributions. The experiments spanned a wide range of scenarios, including univariate and multivariate artificial datasets with known noise characteristics, as well as real-world applications like power plant output prediction and electricity demand forecasting. The models used were standard multilayer perceptrons (MLPs) trained via Bayes-by-Backprop, with performance evaluated using metrics such as Mean Squared Error (MSE) and the Continuous Ranked Probability Score (CRPS).

The study demonstrates that the choice of likelihood distribution significantly impacts BNN performance. Contrary to the intuition that the likelihood should match the specific noise distribution of the data, the heavy-tailed Student's t-distribution consistently outperformed the Gaussian baseline across diverse datasets and model architectures. This performance boost was observed regardless of the data's true noise distribution, suggesting that the Student's t-distribution serves as a more robust default choice for BNNs. Furthermore, the authors noted that this approach is easy to implement and can occasionally lead to faster training convergence.