Noor Islam S. Mohammad
4 min
Stacked generalization is a standard technique for improving the accuracy and robustness of deep learning models by combining heterogeneous backbones. However, practitioners often face two major issues: prediction-space multicollinearity, which makes the meta-learner unstable, and calibration collapse, where stacking multiple models leads to worse uncertainty estimates than a single model. Existing methods like ridge regularization or greedy selection address these problems only partially. The author introduces CORE-STACK+, a meta-learning pipeline that replaces four common heuristics with principled, deterministic preconditioning steps: (i) a kernelized redundancy filter using Centered Kernel Alignment (CKA) to detect non-linear dependencies; (ii) a differentiable meta-feature gate for sample-conditioned attention; (iii) a spectrum-adaptive ridge penalty derived from Marchenko-Pastur signal-noise decomposition; and (iv) a Laplace-approximate Bayesian blender to replace simple inverse-RMSE heuristics.
CORE-STACK+ consistently outperforms existing ensemble methods across six diverse vision benchmarks. On ImageNet-1K, it delivers a +1.8% improvement in top-1 accuracy while reducing the number of retained models by 35–57% and lowering inference FLOPs by up to 41%. The method also significantly improves Expected Calibration Error (ECE) by 2.1x without requiring post-hoc temperature scaling. Furthermore, the author provides the first PAC-Bayes excess-risk bound that explicitly couples prediction-space redundancy to meta-learner capacity, providing a theoretical foundation for why reducing the Gram matrix condition number leads to better generalization.
This work reframes ensemble meta-learning as a conditioning problem rather than just an aggregation task. By providing a systematic way to prune redundant models and stabilize the meta-learner, CORE-STACK+ enables the deployment of high-performance, uncertainty-aware vision systems on resource-constrained hardware. The ability to achieve state-of-the-art accuracy and calibration without expensive post-hoc corrections or massive ensembles makes it a highly practical tool for safety-critical applications like autonomous driving and medical imaging.
Stacking heterogeneous vision backbones (CNNs, ViTs, and hybrids) is the de facto recipe for accuracy, calibration, and robustness, yet two coupled pathologies limit its returns. Prediction-space multicollinearity ill-conditions the meta-learner's Gram matrix, inflating weight variance and producing brittle solutions on a thin manifold. Calibration collapse compounds constituent miscalibration through naive linear stacking, so adding more models can hurt expected calibration error (ECE). Existing remedies, ridge regularization, greedy selection, model soups, and SWAG address at most one of these issues, and none jointly target conditioning and calibration in heterogeneous prediction pools. We introduce CORE-STACK+, a preconditioning pipeline with four components: (i) a kernelized redundancy filter that removes non-linear inter-model dependencies invisible to Pearson correlation, using Centered Kernel Alignment (CKA) [23]; (ii) a $<15$K-parameter differentiable meta-feature gate that learns per-sample attention over ensemble statistics; (iii) a spectrum-adaptive Ridge penalty $lambda^{star}=lmax(Chat)/SNR(Chat)$ derived from a Marchenko-Pastur signal-noise decomposition, eliminating nested cross-validation; and (iv) a Laplace-approximate Bayesian blender replacing inverse-RMSE heuristics. We prove a PAC-Bayes excess-risk bound that, for the first time, jointly accounts for prediction-space redundancy and meta-learner capacity. Across six benchmarks, CORE-STACK+ delivers $+1.8\%$ top-1 on ImageNet-1K, $-4.2$ mCE on ImageNet-C, $+0.9$ mIoU on ADE20K, and $+1.3$ AP on COCO, while reducing retained models by 35-57% and inference FLOPs by up to $41%$. ECE improves $2.1\times$ over deep ensembles without post hoc temperature scaling.
Alex: [voice brightening] So the Bayesian blender is functioning as a safeguard against overconfidence, rather than a separate calibration step bolted on afterward?
Sam: [quiet confidence] Precisely — it's built into the weighting itself. And on top of that, they use what's called a spectrum-adaptive ridge penalty, derived from a Marchenko-Pastur decomposition of the Gram matrix's eigenvalue spectrum. Practically, that means the regularization strength is set analytically, matched to the signal-to-noise ratio of the spectrum, rather than tuned by nested cross-validation. That's the piece that removes the need for the expensive search most stacking pipelines rely on — it's a closed-form solution instead.
Alex: [reflective] If the pruning step is doing real work, does the ensemble size stabilize on its own as you throw more candidate models at it — does it just start ignoring the junk? [[RP_SECTION:self-regulating-ensemble-size|Self-Regulating Ensemble Size]]
Sam: [steady, grounded] That's the main empirical result worth weighing carefully. As the candidate pool grows, the CKA filter consistently retains only around forty to fifty percent of the models — the rest get pruned as redundant, regardless of how large the initial pool is. That's the load-bearing finding: it shows the system is self-regulating rather than needing the pool size hand-tuned for each new set of base models.
Alex: [slower, processing] That's a meaningful reduction in complexity for a lot less manual tuning. But does the CKA projection itself introduce any bias into what survives? [[RP_SECTION:limitations-and-future-testing|Limitations and Future Testing]]
Sam: [measured] Every projection does — that's the honest caveat here. Filtering on kernel alignment makes an implicit assumption about what "redundant" means, and a careful referee would want to see how sensitive the forty-to-fifty percent retention figure is to the kernel choice and the Nyström approximation's rank, rather than just the aggregate calibration numbers. The paper frames the closed-form regularization and the retention behavior as the two pillars; the kernel choice is the part I'd want stress-tested further before treating it as settled.
Alex: [reflective] So the case for CORE-STACK+ rests on those two pillars — the retention behavior holding up as pool size grows, and the calibration gains under shift — with the kernel sensitivity as the open question underneath it.
Sam: [quiet, settling] That's a fair summary of where the weight sits.
Alex: If you want the figures and the method choices we didn't walk through, you can generate a deep dive of this paper — the paper has the rest either way.
Sam: Thanks for listening.