SSM Adapters via Hankel Reduced-order Modeling: Injection Site Determines Task Suitability in Long-Context Fine-Tuning

Standard parameter-efficient fine-tuning (PEFT) methods like LoRA are position-independent, meaning they process each token in isolation and lack the ability to maintain a persistent state across long sequences. This paper investigates whether inserting a recurrent state space model (SSM) adapter into frozen transformer models can improve performance on tasks requiring sequential state accumulation, such as long-context reasoning and state tracking.

The author introduces the Hankel Reduced-order Model (HRM) adapter, a residual module designed to be injected parallel to the MLP blocks of a frozen transformer. Unlike LoRA, which computes a static linear transformation, the HRM adapter uses a recurrent hidden state to integrate information from prior tokens. To ensure computational efficiency, the author enforces time-invariance on the system matrix, allowing the recurrence to be computed via an exact FFT-based parallel scan. The adapter is initialized using Balanced Truncation of empirical Hankel Grammians, a control-theoretic technique that ensures the model is compressible to a minimal state dimension while maintaining stability.

In iso-parametric evaluations on the Mistral-7B model, the HRM adapter consistently outperformed LoRA variants across various benchmarks. Specifically, it achieved significant gains on long-context tasks, including a 34.8% relative accuracy improvement on QuALITY and a 71.6% relative ROUGE-1 improvement on QMSum. The adapter also demonstrated superior performance in synthetic state-tracking tasks and character-level language modeling. Gate analysis suggests that the HRM adapter successfully learns to modulate recurrence, providing a robust, memory-aware alternative to static low-rank adaptation.