Combining pretrained models is a common strategy for building AI systems, whether by aligning a vision encoder with a language model or merging fine-tuned checkpoints. However, finding compatible pairs typically requires costly empirical search, as compatibility is known only after alignment or merging. We propose COMPASS (COMPatibility ASsessment Score), a theoretically motivated diagnostic that estimates whether a model pair will be compatible prior to alignment or merging. Building on the observation that representational convergence is most robust at the level of local neighborhoods, COMPASS decomposes model compatibility into two cheap-to-compute axes: a pair-specific ease, defined as the $k$-NN overlap between the two encoders, and a paradigm-level ceiling, capturing the maximum overlap their training paradigms allow. Together, these two axes provide a ranking of candidate model pairs and prescribe how to act on each pair: (i) prioritize pairs that already agree and whose paradigms are compatible, (ii) invest additional compute on pairs with compatible paradigms but low initial agreement, (iii) switch the pretraining paradigm when it cannot support sufficient agreement, or (iv) deprioritize pairs that score poorly on both axes. We apply COMPASS to $72$ cross-modal alignment experiments and $392$ task-arithmetic model merges and show that compatibility estimates by COMPASS strongly correlate with downstream success, achieving average Pearson correlation of $0.91$ on cross-modal alignment tasks and $0.70$ on model merging.
Sam: You can screen pretrained model pairs before spending the compute to combine them. This study proposes a cheap compatibility check that predicts which pairs are more likely to work together.
Alex: That could change how people choose models. Is this about linking models with different jobs, or combining versions trained for different tasks?
Sam: Both, within the settings tested. It’s an arXiv method preprint by Fabian Gröger, Petar Damjanović, and Maria Brbić. Their method, COMPASS, stands for Compatibility Assessment Score.
Alex: Usually, you discover whether models cooperate by doing the expensive experiment. The useful claim here is that you can make a better choice beforehand.
Sam: That’s the intended payoff for researchers building image-and-language systems or combining task-specific models. Instead of trying every pairing, you rank candidates from a small sample. We’ll unpack what the score measures, how well it predicts success, and where its guidance stops.
Alex: Start with what was missing. Researchers already compare model representations, meaning their internal encodings of examples. What makes this more than another similarity measure?
Sam: The authors build on a specific distinction. The Platonic Representation Hypothesis proposed that capable models converge toward a shared global representation. The Aristotelian Representation Hypothesis challenged that picture: agreement appears more reliably in local neighborhoods.
Alex: So models needn’t arrange everything into the same overall shape. They might still agree about which examples belong near one another.
Sam: That’s the useful observation. Take an example in a small probe dataset. Find its ten nearest neighbors under each model, then measure how many neighbors the models share. Average that overlap across the examples.
Alex: For images and text, those aren’t identical inputs. How do you make the comparison meaningful?
Sam: The probe contains paired images and captions. The image model encodes the images, and the language model encodes the corresponding text. You compare which paired examples become neighbors in each space, without requiring matching dimensions or identical global geometry.
Alex: That gives you agreement now. But the purpose of alignment is to improve agreement. Why should the starting point tell us what happens afterward?
Sam: That question motivates the first component, called ease. It measures the particular pair’s initial neighborhood overlap. The authors give a conditional theoretical argument: if training makes progress and reduces the gap between representations, starting closer improves a finite-budget guarantee.
Alex: Conditional matters there. They haven’t proved that high initial agreement makes every training procedure succeed.
Sam: They explicitly call the theory supporting motivation, not a universal guarantee. The assumptions concern reaching a target and shrinking the gap during training. Those assumptions may fail, and the resulting bound can be uninformative.
Alex: There’s another possible problem. Two models could start close but be trained to preserve different things. Does the score distinguish a difficult start from a poor destination?
Sam: That’s the second component, called ceiling. It estimates how much agreement the training approaches support for the application. Their other theoretical result says that differing target neighborhoods can limit agreement, even when each model learns its own target well.
Alex: Those targets aren’t directly observable. What do they actually calculate?
Sam: They use an empirical proxy, and its definition changes with the application. For image-text alignment, they average a vision encoder’s overlap across a reference library of language encoders. That gives a broader signal than its agreement with one particular partner.
Alex: A vision model trained alongside language might therefore have a different outlook from one trained to reconstruct pixels.
Sam: That’s their example. Contrastive vision-language models learn to bring corresponding images and text together, and have higher alignment ceilings here. Masked autoencoders, trained through image reconstruction, have lower ceilings. COMPASS multiplies ease and ceiling to rank pairs that score well on both.
Alex: And if someone wants advice rather than just a ranked list, what do those components tell them?
Sam: High ease and high ceiling mean “act now.” Low ease with high ceiling means “budget-limited”: consider more data, optimization, or a stronger partner. High ease with low ceiling suggests changing the training paradigm. Low scores on both mean deprioritize the pair.
Alex: Those labels sound firmer than a ranking. Are the boundaries supposed to mark real discontinuities?
Sam: No. They split each axis at its median to make an interpretable summary. The continuous product is the main output. “Incompatible” means a poor prospect under the tested recipe and realistic budgets, not something that could never work.
Alex: Let’s get to the evidence. What did they actually combine, and how did they judge success?
Sam: They evaluated seventy-two vision-language pairs. Both encoders stayed frozen while an adapter mapped their outputs into a shared space. Success meant retrieving the corresponding caption as the top result, tested on Flickr30k and COCO.
Alex: Did COMPASS track those outcomes better than a conventional global similarity measure?
Sam: On COCO, its Pearson correlation with retrieval performance was point nine one. The global comparison, centered kernel alignment, reached point six eight. That comparison measures overall representational similarity rather than shared local neighbors.
Alex: That’s strong association, but selection is the practical task. Did the ranking lead them toward good candidates?
Sam: In the tested alignment pool, COMPASS put the best pair first. The global measure also selected that pair first, so that result wasn’t unique. COMPASS’s advantage became clearer when ranking a broader shortlist.
Alex: What about combining task-specific models? That changes weights directly, rather than learning a bridge between frozen encoders.
Sam: They tested task arithmetic, which combines task-specific changes to a shared model’s weights. Here they averaged checkpoints fine-tuned from the same base model on different classification tasks. They evaluated three hundred ninety-two pairwise merges.
Alex: The outcome should then be how much of each model’s original ability survives, not just the merged model’s raw accuracy.
Sam: They measured average accuracy retained relative to the separate checkpoints, across each pair’s original tasks. COMPASS correlated with retention at point seven zero. The global similarity baseline reached point two five.
Alex: Does the ceiling mean the same thing in that setting?
Sam: The interpretation is still application-level compatibility, but the calculation differs. For merging, the proxy measures how distinct the pretrained base model’s features are from a reference vision library. Ease still measures agreement between the specific fine-tuned checkpoints.
Alex: So a model family that’s favorable for image-text alignment needn’t be favorable for classification merges.
Sam: The study finds that reversal. Contrastive vision-language models favor alignment but fare poorly in these merges. Classification-trained and self-distillation models show the reverse pattern. Compatibility belongs to the transfer operation, not to a universally good model family.
Alex: How much should we trust this beyond those broad family differences?
Sam: Alignment agreement remained predictive within individual vision families, and additional adapter types gave consistent rankings on a subset. Merging was less convincing within a single backbone than across the pooled library. Much of the retention variation lay between backbones.
Alex: That makes it useful screening evidence, but weaker evidence for fine distinctions among closely related merges. What’s the biggest obstacle to applying it elsewhere?
Sam: The ceiling has no universal formula. It needs application-specific knowledge and a sensible reference library. More data, larger adapters, or a different objective could also rescue pairs labeled incompatible. These experiments support a finite-budget diagnostic, not a law of model compatibility.
Alex: Who should read the full document, and where should they start?
Sam: Researchers selecting vision-language pairs or merging classification checkpoints should start with “Cheap-to-compute scores for downstream problems.” Then read “Operational computation of ease and ceiling” alongside “Limitations.” Those sections show both the usable recipe and the choices you must justify. For everyone else: ask whether models suit each other for the job, not whether either model is good alone.
Alex: Keep the job in view.