Rethinking Expert Training for Model Merging with Prompt Learning

Model merging aims to combine multiple domain-specialized experts into a single multi-task model without joint retraining. While most research focuses on improving the merging algorithms themselves (e.g., Task Arithmetic, TIES), this paper investigates whether the expert-training procedure itself can be optimized to produce more mergeable models.

The authors introduce Dual-Tuned Experts (DTEs), a two-stage training strategy. In the first stage, they perform prompt learning, where only textual context tokens are optimized while the vision and text encoders remain frozen. In the second stage, the vision encoder is fine-tuned while keeping the previously learned prompts fixed. This approach aims to offload part of the task-specific adaptation to the prompts, thereby reducing the magnitude of the updates required in the vision backbone.

The authors demonstrate that prompt-based adaptation serves as a strong baseline for multi-task inference, often rivaling complex merging methods. By combining this with vision fine-tuning, DTEs produce experts that exhibit significantly less representation drift and smaller parameter updates compared to standard full fine-tuning. Experiments across various CLIP architectures and merging methods (such as TIES and Iso-C) show that DTEs consistently improve the performance of the resulting merged models, even when combining heterogeneous sets of experts.

This work shifts the focus of model merging from purely algorithmic improvements to the design of the expert-training pipeline. By showing that the way experts are trained directly impacts their compatibility in parameter space, the authors provide a simple, effective strategy to enhance the performance of existing merging techniques without requiring architectural changes to the merging process itself.