ORBIT: Training-Free Multi-Attribute Behavioral Steering via Orthogonal Subspace Rotation

As language models are increasingly deployed in assistant roles, users often need to control multiple behavioral attributes simultaneously (e.g., being both concise and empathetic). Existing inference-time steering methods, such as Contrastive Activation Addition (CAA), often fail when applied to multiple attributes because they suffer from geometric interference, where steering vectors either cancel each other out or cause one attribute to dominate the others. This paper investigates how to achieve balanced, multi-attribute control without the need for retraining or complex classifier-based interventions.

The authors introduce ORBIT (Orthogonal Rotation-Based Intervention Technique), a geometric framework that treats multi-attribute steering as a rotation problem. Instead of summing independent steering vectors, ORBIT first constructs a shared, low-dimensional subspace that captures all target attributes using singular value decomposition (SVD). It then applies a single, norm-preserving rotation within this joint subspace toward a combined target direction. To ensure precision, the method employs adaptive per-token gating, which only applies steering to tokens that are misaligned with the desired behavior, and an optional additive boost to strengthen weak signals.

ORBIT consistently outperforms existing training-free baselines across three different model families (Llama-3.2, Qwen-2.5, and Llama-3.1). Unlike additive methods that often degrade output coherence or cause "attribute dominance" (where one trait improves at the expense of others), ORBIT maintains high coherence and achieves positive geometric gain across all tested configurations. This indicates that ORBIT successfully improves all targeted attributes simultaneously. The authors also introduce TRAITFACTORY, a new benchmark focusing on deep-seated behavioral tendencies, which demonstrates that ORBIT remains effective even for complex personality traits that are typically resistant to simple additive steering.