Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learning

Federated fine-tuning of large pre-trained models using Low-Rank Adaptation (LoRA) often suffers from instability when clients have heterogeneous data. The authors identify that this instability stems from data-parameter interference: when adapters are aggregated, an adapter trained on one client's data is applied to another client's representations, causing unpredictable model behavior. The study asks how to coordinate these LoRA updates to minimize this interference without violating federated learning privacy constraints.

The authors propose Dynamic Subspace Boosting (Dysco), a method that shifts the focus of federated LoRA from simple parameter averaging to dynamic subspace allocation. In each round, clients identify 'activation-insensitive' subspaces—directions in the feature space that produce minimal response on their local data—and transmit only these bases to the server. The server then constructs client-specific merged subspaces that maximize compatibility with other clients' insensitive directions. To account for representation drift, Dysco uses a boosting mechanism that stacks new LoRA blocks on top of previously learned ones, allowing the model to adapt to changing data distributions over time.

Theoretical analysis shows that Dysco provides a tighter upper bound on aggregation error compared to standard federated LoRA methods by structurally reducing the interference term. Empirically, Dysco significantly outperforms existing baselines, including FedAvg-LoRA and other federated LoRA variants. In synthetic tasks, it reduced training loss by up to 9 times. On real-world clinical note classification (MIMIC-IV) using Llama-3.2-1B, Dysco improved performance across all tested federated learning algorithms, achieving a 4.3% gain over FedAvgM, while adding only 0.9% to the total wall-clock time.