ResearchPod Summary
Multi-view learning traditionally assumes a small, fixed set of sensors or features. However, modern foundation-model hubs offer a large, heterogeneous pool of pretrained encoders, turning multi-view fusion into a combinatorial selection problem: which encoders should be combined, and how many? This paper investigates why downstream performance is non-monotonic when more foundation models are fused, identifying that later views can be redundant or misaligned with the target task. To solve this without the prohibitive cost of training downstream classifiers for every candidate subset, the authors formulate view-set composition as an ordering problem and propose KAGES (Kernel-Alignment Greedy Encoder Selector).
The authors analyze the failure of the "more is better" assumption through a marginal-utility lens, establishing that the finite-sample test risk is governed by a trade-off between the Bayes gain and excess-risk costs. When an encoder is added along a fixed path, its marginal utility can cross from positive to negative. The paper formalizes this via an order-specific gain-cost crossing theorem, showing that accuracy peaks at an intermediate number of encoders (typically 3 to 5 on many datasets) and subsequently declines. Furthermore, empirical evaluations across diverse recognition regimes—such as fine-grained objects, textures, and geo-spatial scenes—demonstrate that this peak-then-decline phenomenon is structural, and that a single task-aligned encoder often outperforms indiscriminate full fusion.
To construct an optimal encoder ordering without downstream classifier training, KAGES evaluates candidate encoders using a label-aware, set-level kernel-alignment objective. By measuring the alignment between the joint representation kernel of the selected subset and the label kernel, KAGES can score marginal gains efficiently at the kernel level in quadratic time relative to sample size, independent of encoder dimensionality. Under mild monotonicity and submodularity conditions, the greedy prefixes formed by KAGES enjoy a formal theoretical performance guarantee. Across extensive experiments in image classification, image retrieval, and frozen large-language-model fusion, KAGES consistently outperforms diversity-based baselines and full fusion, approaching the performance of exhaustive oracle selection.
Alex: Welcome to another episode of ResearchPod.
Sam: Today we're looking at a paper called "When More Foundation Models Means Less." The central claim is that in multi-view fusion, performance is order-dependent — and the optimal strategy is to select a compact, task-aligned subset of encoders, not pool every available one.
Alex: So the question isn't how to fuse representations, it's what to fuse. And the answer isn't just "everything."
Sam: Right. The design space has exploded — we've gone from a handful of pretrained models to hubs with dozens of viable encoders. If you just combine everything, performance often saturates or actually declines. That decline is the paper's motivating observation.
Alex: Why would adding more features ever hurt?
Sam: Two failure modes. The first is representation redundancy — a new encoder repeats information the system already has, adding noise without signal. The second is task-model misalignment — an encoder injects cues that are irrelevant to the downstream task, diluting the useful features from encoders that were actually working. Neither failure is obvious from standalone performance. A model that looks strong in isolation can actively hurt a fusion stack.
Alex: So you can't just rank encoders individually and take the top-k. You need to reason about each encoder's marginal contribution given what's already in the stack.
Sam: Exactly. And that's the combinatorial problem the paper is trying to solve — without retraining a classifier for every possible subset. Their solution is KAGES: the Kernel-Alignment Greedy Encoder Selector. It's training-free. It works by greedily adding encoders one at a time, each time selecting the candidate that maximizes the marginal gain in Centered Kernel Alignment — CKA — between the fused feature kernel and the label kernel.
Alex: So CKA is measuring how well the current feature representation aligns with the target labels. What makes this tractable at scale?
Sam: The key insight is that linear kernels are additive. When you concatenate encoder outputs, the combined kernel is just the sum of the individual encoder kernels. That means KAGES can evaluate the marginal utility of any candidate encoder analytically, without touching a downstream predictor. What would otherwise be an exponential search over subsets becomes a sub-second traversal. And the theory provides a submodularity guarantee — the greedy prefix is a provably strong approximation of the optimal subset under certain conditions.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Alex: What conditions? Submodularity guarantees tend to come with fine print.
Sam: They do. The CKA objective isn't strictly monotone or submodular in general — the denominator can sometimes outweigh the numerator's gain, and there are cross-overlap terms the theory doesn't fully capture. The guarantee is conditional on the encoder pool satisfying certain monotonicity and submodularity-ratio conditions, and there's no universal proof that empirical frozen-encoder pools satisfy these. So it's a heuristic with strong theoretical motivation, not a formal proof for every possible encoder set.
Alex: That's an important distinction. It works well in practice, but the theoretical coverage has limits. What does the empirical picture look like?
Sam: The main finding is that KAGES consistently selects compact subsets — typically two to four encoders — that match or outperform the full pool. The load-bearing result is that the full ensemble is rarely optimal, and the performance curve is non-monotone: it rises, peaks, then falls as you keep adding encoders. That inverted-U pattern is what the paper's central claim rests on.
Alex: And is there a pattern in which encoders actually survive the selection?
Sam: There is, and it's one of the more interpretable findings. KAGES consistently deprioritizes generative and task-expert encoders — likely because they're structurally misaligned with classification objectives — and gravitates toward self-supervised encoders. That's not a design constraint; it's an emergent preference from the CKA objective. Which suggests the alignment signal is genuinely capturing task relevance, not just feature diversity.
Alex: That's a meaningful sanity check. If the method were just selecting for diversity, you'd expect it to retain more of the heterogeneous encoders.
Sam: Right. The fact that it rejects generative models systematically is evidence that the kernel alignment signal is doing real work. The supporting ablations confirm this — when you swap in random selection, or rank by individual CKA score rather than marginal gain, performance drops. The marginal, greedy framing is doing something the simpler alternatives can't replicate.
Alex: Where would a careful referee push back?
Sam: A few places. First, all the benchmarks are classification tasks. The method's behavior on regression, dense prediction, or structured output tasks is untested — and the submodularity argument may not transfer cleanly. Second, the evaluation uses frozen encoders throughout. If you allow fine-tuning, the interaction effects between encoders change, and it's not clear the greedy kernel-level selection would remain optimal. Third, the paper doesn't address the computational cost of running multiple encoders in parallel at inference time — a real deployment consideration that the training-free framing sidesteps.
Alex: So the practical claim is strongest in the frozen, classification setting.
Sam: Which covers a lot of real use cases, but not all of them. The honest read is that this is a precise, well-supported argument within that regime — not a universal claim about multi-modal fusion. The contribution is identifying a tractable proxy for subset quality and showing empirically that the greedy approximation holds up. The scope just needs to be read carefully.
Alex: The broader implication being that model hub curation is a first-class design problem — not something you can defer by just using everything available.
Sam: That's the takeaway. As model hubs keep growing, the selection problem only compounds. KAGES is a computationally cheap entry point. The open questions are whether the same kernel-alignment logic extends to tasks with less clean label structure, and whether selection criteria need to adapt dynamically to specific queries rather than being fixed at deployment time. Both feel like natural next steps.
Alex: A lot of interesting ground left to cover. Thanks for walking through it — and thanks to everyone listening to ResearchPod.