Spectral Transformation for Layer-wise Global Rank Discovery in Federated LoRA for Vision Transformers

In federated learning, fine-tuning Vision Transformers (ViTs) with Low-Rank Adaptation (LoRA) is a standard approach to reduce communication costs. However, aggregating these local adapters across clients is difficult: simple averaging introduces noise, while stacking adapters increases server overhead and communication costs. This paper addresses the challenge of performing exact, communication-efficient aggregation while automatically determining the optimal rank for each layer without manual hyperparameter tuning.

The authors propose SpecTraL, a framework that treats federated LoRA aggregation as a spectral problem. Instead of performing computationally expensive Singular Value Decomposition (SVD) on large stacked matrices, SpecTraL uses Householder QR decomposition to compress the stacked adapters into a small, core interaction matrix. By applying the ScreeNOT estimator—a tool from Random Matrix Theory—the server can analytically separate the task-relevant consensus signal from the incoherent noise caused by non-IID data. This allows the system to determine the intrinsic dimensionality of the update for every layer individually. Finally, a padding-aware initialization strategy ensures that clients can resume local training with their full capacity while preserving the denoised global signal.

SpecTraL solves the fundamental trade-off between communication efficiency and model accuracy in federated LoRA. By replacing heuristic-based rank selection with a statistically grounded method, it removes the need for exhaustive hyperparameter searches that are typically required to find the right "energy threshold" for different models and datasets. The method demonstrates superior accuracy-communication trade-offs on standard benchmarks like DomainNet and NICO++, proving that principled spectral denoising is more effective than fixed-rank or energy-based aggregation strategies.