Yibo Liu, Bowen Jiang
6 min
Multi-view learning traditionally assumes a small, fixed set of sensors or features. However, modern foundation-model hubs offer a large, heterogeneous pool of pretrained encoders, turning multi-view fusion into a combinatorial selection problem: which encoders should be combined, and how many? This paper investigates why downstream performance is non-monotonic when more foundation models are fused, identifying that later views can be redundant or misaligned with the target task. To solve this without the prohibitive cost of training downstream classifiers for every candidate subset, the authors formulate view-set composition as an ordering problem and propose KAGES (Kernel-Alignment Greedy Encoder Selector).
The authors analyze the failure of the "more is better" assumption through a marginal-utility lens, establishing that the finite-sample test risk is governed by a trade-off between the Bayes gain and excess-risk costs. When an encoder is added along a fixed path, its marginal utility can cross from positive to negative. The paper formalizes this via an order-specific gain-cost crossing theorem, showing that accuracy peaks at an intermediate number of encoders (typically 3 to 5 on many datasets) and subsequently declines. Furthermore, empirical evaluations across diverse recognition regimes—such as fine-grained objects, textures, and geo-spatial scenes—demonstrate that this peak-then-decline phenomenon is structural, and that a single task-aligned encoder often outperforms indiscriminate full fusion.
To construct an optimal encoder ordering without downstream classifier training, KAGES evaluates candidate encoders using a label-aware, set-level kernel-alignment objective. By measuring the alignment between the joint representation kernel of the selected subset and the label kernel, KAGES can score marginal gains efficiently at the kernel level in quadratic time relative to sample size, independent of encoder dimensionality. Under mild monotonicity and submodularity conditions, the greedy prefixes formed by KAGES enjoy a formal theoretical performance guarantee. Across extensive experiments in image classification, image retrieval, and frozen large-language-model fusion, KAGES consistently outperforms diversity-based baselines and full fusion, approaching the performance of exhaustive oracle selection.
Foundation-model hubs turn multi-view fusion into a selection problem: from a large heterogeneous encoder pool, which views should be fused, and how many? We show that downstream performance is non-monotonic in the number of fused encoders; later views can be redundant or task-misaligned, causing accuracy to saturate or decline. We formalise this setting as view-set composition and propose KAGES (Kernel-Alignment Greedy Encoder Selector), a label-aware method that orders frozen encoders by their marginal gain in centred kernel-target alignment. KAGES requires no downstream classifier training during selection, evaluates each candidate in $\mathcal{O}(n^2)$ time independent of encoder dimension, and admits a conditional $(1-e^{-γ})$ prefix-wise guarantee under monotonicity and a positive submodularity ratio. Across five recognition regimes and low-shot, larger-pool, and full-data protocols, KAGES improves average AULC over full fusion by 3.9, 5.8, and 3.3 points, respectively, and exceeds DPP and facility-location selection in average AULC. Image retrieval exhibits later, task-dependent saturation along the KAGES ordering, while peak-then-decline reproduces in frozen-LLM fusion. These results show that effective large-pool fusion depends on selecting a compact, task-aligned set of views rather than indiscriminately fusing more encoders.
Alex: That's an important distinction. It works well in practice, but the theoretical coverage has limits. What does the empirical picture look like?
Sam: The main finding is that KAGES consistently selects compact subsets — typically two to four encoders — that match or outperform the full pool. The load-bearing result is that the full ensemble is rarely optimal, and the performance curve is non-monotone: it rises, peaks, then falls as you keep adding encoders. That inverted-U pattern is what the paper's central claim rests on.
Alex: And is there a pattern in which encoders actually survive the selection?
Sam: There is, and it's one of the more interpretable findings. KAGES consistently deprioritizes generative and task-expert encoders — likely because they're structurally misaligned with classification objectives — and gravitates toward self-supervised encoders. That's not a design constraint; it's an emergent preference from the CKA objective. Which suggests the alignment signal is genuinely capturing task relevance, not just feature diversity.
Alex: That's a meaningful sanity check. If the method were just selecting for diversity, you'd expect it to retain more of the heterogeneous encoders.
Sam: Right. The fact that it rejects generative models systematically is evidence that the kernel alignment signal is doing real work. The supporting ablations confirm this — when you swap in random selection, or rank by individual CKA score rather than marginal gain, performance drops. The marginal, greedy framing is doing something the simpler alternatives can't replicate.
Alex: Where would a careful referee push back?
Sam: A few places. First, all the benchmarks are classification tasks. The method's behavior on regression, dense prediction, or structured output tasks is untested — and the submodularity argument may not transfer cleanly. Second, the evaluation uses frozen encoders throughout. If you allow fine-tuning, the interaction effects between encoders change, and it's not clear the greedy kernel-level selection would remain optimal. Third, the paper doesn't address the computational cost of running multiple encoders in parallel at inference time — a real deployment consideration that the training-free framing sidesteps.
Alex: So the practical claim is strongest in the frozen, classification setting.
Sam: Which covers a lot of real use cases, but not all of them. The honest read is that this is a precise, well-supported argument within that regime — not a universal claim about multi-modal fusion. The contribution is identifying a tractable proxy for subset quality and showing empirically that the greedy approximation holds up. The scope just needs to be read carefully.
Alex: The broader implication being that model hub curation is a first-class design problem — not something you can defer by just using everything available.
Sam: That's the takeaway. As model hubs keep growing, the selection problem only compounds. KAGES is a computationally cheap entry point. The open questions are whether the same kernel-alignment logic extends to tasks with less clean label structure, and whether selection criteria need to adapt dynamically to specific queries rather than being fixed at deployment time. Both feel like natural next steps.
Alex: A lot of interesting ground left to cover. Thanks for walking through it — and thanks to everyone listening to ResearchPod.