Manifold-Constrained Hyper-Connections for Parameter-Efficient Finetuning

This paper investigates whether Manifold-Constrained Hyper-Connections (mHC)—a generalization of residual connections originally designed for pre-training—can be effectively repurposed as a parameter-efficient finetuning (PEFT) method for frozen Transformer backbones. The authors explore whether modifying how representations are routed across residual streams and depths provides a distinct and complementary axis for model adaptation compared to traditional weight-based PEFT methods like LoRA.

The researchers wrap the OLMo-2 backbone (at 1B and 7B scales) with mHC modules, which replace standard residual connections with learned routing modules. These modules use a doubly stochastic constraint (the Birkhoff polytope) to ensure that the residual stream remains stable during training. The authors evaluate both static variants (fixed routing per layer) and dynamic variants (input-dependent routing) and compare them against standard PEFT baselines. They also test the combination of mHC with LoRA to determine if residual routing provides synergistic benefits.

The study reveals that mHC is a viable PEFT mechanism, but its role in finetuning differs from its role in pre-training. Specifically, fixing the residual mixing matrix to identity often improves performance, suggesting that preserving the original residual structure is crucial during adaptation. While mHC does not consistently outperform LoRA as a standalone method, combining mHC with LoRA at matched parameter budgets leads to improved language-modeling loss and task-specific gains. This indicates that residual routing is a distinct and promising axis for PEFT that can augment existing techniques.