Training-free Task Classification for Multi-Task Model Merging

As foundation models become ubiquitous, researchers often fine-tune them for various downstream tasks. Merging these task-specific experts into a single, efficient model is a major goal, but static merging often suffers from parameter interference, where different tasks conflict. While dynamic merging—which uses input-adaptive routing—can mitigate this, existing methods typically require either expensive additional training or prior knowledge of the task identity for every input. This paper asks: can we achieve high-performance dynamic model merging without training a router or knowing the task ID at inference time?

To solve this, the authors propose Singular-vector-based Manifold (SiM) classification. The core idea is to treat task identification as a geometric problem. For each task, the authors use a small support set (32 examples) to compute a low-rank manifold approximation using Singular Value Decomposition (SVD). During inference, the system calculates the projection residual of an input feature onto each task's manifold. The input is then routed to the task expert whose manifold it aligns with most closely (i.e., the one with the smallest residual). This approach is entirely training-free and requires no task labels during inference.

SiM effectively bridges the performance gap between merged models and individual task experts. Because it is a plug-and-play framework, it integrates seamlessly with existing subspace-based compression methods, allowing for memory-efficient storage of task experts. Experiments across computer vision and natural language processing benchmarks show that SiM consistently outperforms previous dynamic merging methods that rely on additional training or known task identities, while simultaneously reducing the memory footprint and avoiding the need for complex router training.