Learning to Recover Task Experts from a Multi-Task Merged Model

As foundation models are fine-tuned for an increasing number of downstream tasks, storing individual experts becomes prohibitively expensive. While model merging attempts to consolidate these experts into a single unified model, static merging often results in parameter interference, where task-specific updates collide and degrade performance. The authors investigate how to recover the high-fidelity performance of individual experts from a single merged checkpoint without storing redundant expert components at inference time.

The authors propose Recover Task eXpert (ReTeX), a framework that treats parameter interference as a perturbation that can be modeled as an affine transformation. By approximating these perturbations as additive offsets, ReTeX learns to predict task-specific offsets that, when added to the merged model's weights, reconstruct the original expert's behavior. To handle scenarios where the task identity is unknown, the authors introduce a router-free identification method. This method computes SVD-based subspace signatures for each task offline; at inference, it identifies the task by selecting the subspace that yields the smallest projection residual for a given input. This approach avoids the need for training complex routers or storing multiple expert checkpoints.

ReTeX demonstrates significant performance gains across both vision and NLP domains. In experiments with CLIP backbones, ReTeX consistently recovers over 95% of individual-expert performance, even as the number of tasks scales up to 30. Unlike dynamic merging baselines that require storing multiple components or training heavy routers, ReTeX operates as a lightweight, plug-and-play module. Furthermore, the authors observe that the offset prediction mechanism allows for emergent adaptive interpolation, enabling the model to generalize effectively to unseen or out-of-distribution tasks by interpolating between known expert knowledge.