NPBoost: Neural Processes with Gradient-Boosted Fixed Effects

Neural Processes (NPs) are meta-learning models that adapt to new tasks using small context sets. While most research focuses on improving the neural network architecture, this paper proposes a structural decomposition inspired by mixed-effects models. The authors introduce NPBoost, which splits the response variable into two components: a shared fixed-effects function, represented by a gradient-boosted tree ensemble, and task-specific random effects, represented by a latent NP. The two components are trained jointly using a novel boosting algorithm that alternates between updating the NP and fitting the tree ensemble to the residual structure.

NPBoost consistently outperforms standard NPs in both synthetic and real-world tabular meta-learning tasks. The decomposition is particularly effective when the shared structure across tasks contains discontinuities or irregular patterns that boosted trees can capture more efficiently than standard neural networks. In synthetic experiments, NPBoost achieved lower RMSE and CRPS than standard NPs. On real-world datasets, NPBoost reduced RMSE in seven out of eight benchmark scenarios. The authors also demonstrate that their cross-fitting approach for gradient estimation provides unbiased updates, ensuring the model remains robust during training.

This work bridges the gap between traditional statistical mixed-effects models and modern neural meta-learning. By explicitly separating shared population-level structure from task-specific stochastic variation, NPBoost provides a more interpretable and effective framework for tabular data. It demonstrates that architectural complexity is not the only path to better meta-learning; leveraging the strengths of tree-based models for shared structure can significantly improve performance on tabular tasks where neural networks may struggle with irregular patterns.