Escaping Low-Dimensional Overlap: Multi-Task Model Merging via High-Dimensional Sparse Disentanglement

Model merging aims to combine multiple task-specific expert models into a single generalist model without additional training. However, this process often suffers from task interference, where conflicting parameter updates from different tasks degrade the performance of the merged model. The authors investigate this interference through the lens of superposition, where functionally distinct features become entangled in the same parameter space, rendering traditional linear decomposition methods insufficient.

The authors propose a framework that projects task vectors into a high-dimensional sparse feature space using Sparse Autoencoders (SAEs). By mapping parameters into this overcomplete space, the model can explicitly disentangle task-specific components before fusion. To avoid the high computational cost of applying this to every layer, they introduce a Group-Ranked Zeroth-Order Optimizer (GR-ZOO). This optimizer uses forward-pass evaluations to identify task-critical layers, ensuring that the computationally intensive SAE-based disentanglement is only applied where it is most needed.

Experiments on Qwen2.5-1.5B and Qwen2.5-7B demonstrate that the proposed method consistently outperforms representative baselines, including Task Arithmetic, TIES-Merge, and DARE, across diverse tasks such as mathematical reasoning, code generation, and instruction following. In highly conflicting four-task scenarios, the method achieves a 2.78% improvement over the strongest baseline, effectively preserving task-specific capabilities while maintaining alignment.

This work provides a robust, training-free solution to the problem of task interference in model merging. By leveraging mechanistic interpretability insights—specifically the role of superposition—the authors offer a more sophisticated way to fuse models than simple parameter arithmetic. This approach is particularly valuable for creating generalist models from specialized experts in resource-constrained environments where full-parameter fine-tuning is prohibitive.