When Skills Meet Safety: Benchmarking and Characterizing the Adaptive Jailbreak Robustness of Skill-Merged LLMs

Model merging—combining task-specific weight vectors into a base model—has become a standard, training-free method for adding capabilities to LLMs. However, this process often degrades safety. Current safety evaluations rely almost exclusively on static refusal tests, where a model is prompted with fixed harmful inputs and scored on its compliance. This paper argues that such tests are misleading because safety alignment is shallow, primarily residing in the first few generated tokens. A merged model may appear perfectly safe under static evaluation while remaining highly vulnerable to adaptive attacks that steer the model past its initial refusal.

The authors introduce SkillSafe-Bench, a controlled, factorial benchmark designed to evaluate merged models across three dimensions: static refusal, adaptive jailbreak robustness, and capability retention. By testing six open-weight base models across five families, the researchers demonstrate that static safety is a poor predictor of adaptive robustness. For instance, models like Gemma-2-9B and Qwen-2.5 appear highly safe under static screening but are jailbroken 60–76% of the time when subjected to semantic template attacks. In contrast, models like Llama-3.1 and Phi-4 maintain significantly higher robustness under the same conditions.

The paper identifies a data-free geometric signal—the overlap between a task vector and the model's safety subspace—that characterizes how merging erodes safety. By isolating the refusal direction (the safety subspace) from the capability components, the authors propose SubSafe-Merge. This technique projects task vectors onto the orthogonal complement of the safety subspace, effectively removing the merge-induced safety erosion while preserving the intended skill capabilities. The authors emphasize that this is a diagnostic and repair tool, not a replacement for rigorous adaptive evaluation.