HyperSafe: Inference-Time Safety Recovery for Fine-Tuned Language Models

Fine-tuning large language models (LLMs) for specific tasks often inadvertently degrades their safety alignment, causing them to become more compliant with harmful requests. Existing defenses either require costly retraining, modify the model's weights—potentially damaging task performance—or rely on generic, model-agnostic classifiers that fail to account for the specific ways a model's internal representations have shifted. The authors ask: can we restore safety in a fine-tuned model without retraining, weight modification, or sacrificing task utility?

HyperSafe introduces a post-hoc, model-specific safety recovery framework. Instead of altering the fine-tuned model, it attaches a lightweight Safe Side Network (SSN) that runs alongside the frozen backbone. The SSN acts as a gatekeeper, performing prompt-level safety classification to route harmful inputs to a refusal response while allowing safe inputs to pass through the original model.

To avoid the need for manual training per checkpoint, the authors use a hypernetwork. This hypernetwork takes the fine-tuned model's "activation fingerprints"—layer-wise hidden state representations extracted from a few calibration prompts—and maps them to the specific parameters of an SSN. This process requires only a single forward pass, making it highly efficient for deployment.

HyperSafe was evaluated on Qwen2-7B and LLaMA-3-8B models across 22 diverse datasets. The results show that while standard LoRA fine-tuning significantly increases the harmful response rate (from ~5% to 19–31%), HyperSafe consistently reduces this rate to below 1% across all held-out checkpoints. Crucially, this safety restoration is achieved while maintaining downstream task accuracy within 1% of the fine-tuned baseline. Compared to model-agnostic filters like Llama Guard 4, HyperSafe demonstrates superior detection performance on zero-shot benchmarks, likely because it is conditioned on the specific representational changes induced by the fine-tuning process.