Fabian Gröger, Petar Damjanović, Maria Brbić
9 min
Abstract
Combining pretrained models is a common strategy for building AI systems, whether by aligning a vision encoder with a language model or merging fine-tuned checkpoints. However, finding compatible pairs typically requires costly empirical search, as compatibility is known only after alignment or merging. We propose COMPASS (COMPatibility ASsessment Score), a theoretically motivated diagnostic that estimates whether a model pair will be compatible prior to alignment or merging. Building on the observation that representational convergence is most robust at the level of local neighborhoods, COMPASS decomposes model compatibility into two cheap-to-compute axes: a pair-specific ease, defined as the $k$-NN overlap between the two encoders, and a paradigm-level ceiling, capturing the maximum overlap their training paradigms allow. Together, these two axes provide a ranking of candidate model pairs and prescribe how to act on each pair: (i) prioritize pairs that already agree and whose paradigms are compatible, (ii) invest additional compute on pairs with compatible paradigms but low initial agreement, (iii) switch the pretraining paradigm when it cannot support sufficient agreement, or (iv) deprioritize pairs that score poorly on both axes. We apply COMPASS to $72$ cross-modal alignment experiments and $392$ task-arithmetic model merges and show that compatibility estimates by COMPASS strongly correlate with downstream success, achieving average Pearson correlation of $0.91$ on cross-modal alignment tasks and $0.70$ on model merging.
Alex: Conditional matters there. They haven’t proved that high initial agreement makes every training procedure succeed.
Sam: They explicitly call the theory supporting motivation, not a universal guarantee. The assumptions concern reaching a target and shrinking the gap during training. Those assumptions may fail, and the resulting bound can be uninformative.
Alex: There’s another possible problem. Two models could start close but be trained to preserve different things. Does the score distinguish a difficult start from a poor destination?
Sam: That’s the second component, called ceiling. It estimates how much agreement the training approaches support for the application. Their other theoretical result says that differing target neighborhoods can limit agreement, even when each model learns its own target well.
Alex: Those targets aren’t directly observable. What do they actually calculate?
Sam: They use an empirical proxy, and its definition changes with the application. For image-text alignment, they average a vision encoder’s overlap across a reference library of language encoders. That gives a broader signal than its agreement with one particular partner.
Alex: A vision model trained alongside language might therefore have a different outlook from one trained to reconstruct pixels.
Sam: That’s their example. Contrastive vision-language models learn to bring corresponding images and text together, and have higher alignment ceilings here. Masked autoencoders, trained through image reconstruction, have lower ceilings. COMPASS multiplies ease and ceiling to rank pairs that score well on both.
Alex: And if someone wants advice rather than just a ranked list, what do those components tell them?
Sam: High ease and high ceiling mean “act now.” Low ease with high ceiling means “budget-limited”: consider more data, optimization, or a stronger partner. High ease with low ceiling suggests changing the training paradigm. Low scores on both mean deprioritize the pair.
Alex: Those labels sound firmer than a ranking. Are the boundaries supposed to mark real discontinuities?
Sam: No. They split each axis at its median to make an interpretable summary. The continuous product is the main output. “Incompatible” means a poor prospect under the tested recipe and realistic budgets, not something that could never work.
Alex: Let’s get to the evidence. What did they actually combine, and how did they judge success?
Sam: They evaluated seventy-two vision-language pairs. Both encoders stayed frozen while an adapter mapped their outputs into a shared space. Success meant retrieving the corresponding caption as the top result, tested on Flickr30k and COCO.
Alex: Did COMPASS track those outcomes better than a conventional global similarity measure?
Sam: On COCO, its Pearson correlation with retrieval performance was point nine one. The global comparison, centered kernel alignment, reached point six eight. That comparison measures overall representational similarity rather than shared local neighbors.
Alex: That’s strong association, but selection is the practical task. Did the ranking lead them toward good candidates?
Sam: In the tested alignment pool, COMPASS put the best pair first. The global measure also selected that pair first, so that result wasn’t unique. COMPASS’s advantage became clearer when ranking a broader shortlist.
Alex: What about combining task-specific models? That changes weights directly, rather than learning a bridge between frozen encoders.
Sam: They tested task arithmetic, which combines task-specific changes to a shared model’s weights. Here they averaged checkpoints fine-tuned from the same base model on different classification tasks. They evaluated three hundred ninety-two pairwise merges.
Alex: The outcome should then be how much of each model’s original ability survives, not just the merged model’s raw accuracy.
Sam: They measured average accuracy retained relative to the separate checkpoints, across each pair’s original tasks. COMPASS correlated with retention at point seven zero. The global similarity baseline reached point two five.
Alex: Does the ceiling mean the same thing in that setting?
Sam: The interpretation is still application-level compatibility, but the calculation differs. For merging, the proxy measures how distinct the pretrained base model’s features are from a reference vision library. Ease still measures agreement between the specific fine-tuned checkpoints.
Alex: So a model family that’s favorable for image-text alignment needn’t be favorable for classification merges.
Sam: The study finds that reversal. Contrastive vision-language models favor alignment but fare poorly in these merges. Classification-trained and self-distillation models show the reverse pattern. Compatibility belongs to the transfer operation, not to a universally good model family.
Alex: How much should we trust this beyond those broad family differences?
Sam: Alignment agreement remained predictive within individual vision families, and additional adapter types gave consistent rankings on a subset. Merging was less convincing within a single backbone than across the pooled library. Much of the retention variation lay between backbones.
Alex: That makes it useful screening evidence, but weaker evidence for fine distinctions among closely related merges. What’s the biggest obstacle to applying it elsewhere?
Sam: The ceiling has no universal formula. It needs application-specific knowledge and a sensible reference library. More data, larger adapters, or a different objective could also rescue pairs labeled incompatible. These experiments support a finite-budget diagnostic, not a law of model compatibility.
Alex: Who should read the full document, and where should they start?
Sam: Researchers selecting vision-language pairs or merging classification checkpoints should start with “Cheap-to-compute scores for downstream problems.” Then read “Operational computation of ease and ceiling” alongside “Limitations.” Those sections show both the usable recipe and the choices you must justify. For everyone else: ask whether models suit each other for the job, not whether either model is good alone.
Alex: Keep the job in view.