Compliance2LoRA: On-Demand Safety Alignment on Arbitrary Policy Subsets via Hypernetwork-Generated LoRA Adapters

As large reasoning models (LRMs) are deployed across diverse user bases, they must adhere to varying sets of safety policies. Traditional approaches—such as training a unique model for every policy combination or using in-context learning—suffer from either combinatorial storage overhead or increased computational costs due to long context windows. This paper asks whether a unified, hypernetwork-based framework can provide on-demand, policy-specific alignment without sacrificing task performance or efficiency.

The authors propose Compliance2LoRA, a framework that uses a hypernetwork to generate LoRA weights conditioned on specific safety policy embeddings. These policies are converted into embeddings and processed through a trainable attention mechanism that allows for dynamic masking. By toggling these masks, the system can activate or deactivate specific safety constraints at inference time. The model is trained using a combination of supervised fine-tuning and Direct Preference Optimization (DPO), utilizing a deliberative alignment strategy that distills reasoning traces into the model to ensure it understands the role of each policy embedding.

Compliance2LoRA successfully achieves on-demand policy compliance on a single base model. Experiments across different model sizes (1.5B and 7B parameters) demonstrate that the framework matches or exceeds the safety performance of traditional baselines while significantly reducing the computational and storage requirements. Furthermore, the model shows strong generalization capabilities, maintaining effective safety alignment even when presented with policy subset combinations that were not explicitly included in the training data.

This work provides a scalable solution for personalizing LRM safety. By decoupling policy compliance from the base model weights, developers can maintain a single, efficient model that adapts to the specific regulatory or user-preference requirements of different environments, effectively solving the combinatorial explosion problem associated with multi-policy alignment.