Rethinking Heterogeneous LLM Merging: A Weighted Model Averaging Perspective

Rethinking Heterogeneous LLM Merging

Model merging has emerged as a powerful technique for combining the capabilities of multiple large language models (LLMs) without the high cost of full retraining. While traditional merging methods work well for homogeneous models—those sharing the same architecture and parameter shapes—they struggle with heterogeneous models that differ in scale or design. This paper investigates whether a minimalist, training-free approach can bridge these differences using only deterministic dimensional adaptation and weighted interpolation.

The Merging Strategy

The authors propose two primary strategies to align models with mismatched parameter spaces:

• Union-style merging: The smaller model is expanded into the larger model's parameter space. New dimensions are initialized to minimize functional disruption (e.g., zero-initializing linear layers and setting normalization scales to neutral values).
• Intersection-style merging: The larger model is truncated to match the smaller model's parameter space by slicing tensors and removing extra layers.

Once the models share the same dimensionality, they are combined using simple convex interpolation. Crucially, this process avoids all complex machinery, such as learned latent spaces, routing modules, or data-driven distillation.

Performance and Limitations

Experiments across the Qwen-family of models (ranging from 3B to 32B parameters) demonstrate that this simple recipe is surprisingly robust. In many cases, the merged model outperforms both source checkpoints by integrating complementary strengths. For instance, union-style merging between Qwen2.5-14B and Qwen2.5-32B yielded a higher average benchmark score than either individual model.

However, the approach is not a universal solution. The authors observe a "seesaw effect," where improvements in one domain (e.g., mathematical reasoning) often come at the expense of performance in another. Additionally, the method is fragile; while small-ratio interpolation provides gains, attempting to balance the models equally often results in catastrophic performance collapse. These findings suggest that while simple averaging is a strong baseline, the fundamental limits of model compatibility remain a significant bottleneck for all merging techniques.