SeFoRA: Sketch-Aggregated Federated Low-Rank Adaptation with Heterogeneous Client Ranks

{ "core_finding": "SeFoRA is a federated learning algorithm for Low-Rank Adaptation (LoRA) that uses linear sketching to aggregate local model updates, effectively resolving the bilinear mismatch problem and enabling training with heterogeneous client ranks without operating in the full model space.", "caveats": "The theoretical convergence guarantees are currently limited to the rank-homogeneous setting, and the algorithm introduces additional communication overhead proportional to the sketch dimensions compared to standard LoRA.", "markdown": "## Research Question\nFederated learning (FL) combined with Low-Rank Adaptation (LoRA) faces two primary technical hurdles: rank heterogeneity (where clients use different adapter sizes, leading to dimension-incompatible matrices) and the bilinear mismatch (where averaging the factor matrices $A_i$ and $B_i$ before multiplication does not equal the average of the full updates $\\Delta W_i$). Existing methods either suffer from high computational costs, operate in the full model space, or fail to resolve the bilinear mismatch. This paper asks: can we construct a rank-heterogeneous LoRA-based FL method that mitigates the bilinear mismatch without operating in the full model space?\n\n## Approach\nThe authors propose SeFoRA, which utilizes linear matrix sketching to compress local updates into compact, fixed-size matrices. Instead of transmitting raw adapter matrices, clients transmit sketches of their local updates. Because the sketching operator is linear, the federator can aggregate these sketches directly to obtain a sketch of the global update. This approach allows the federator to reconstruct a low-rank approximation of the global update using an unsketching algorithm, effectively bypassing the bilinear mismatch while maintaining low computational overhead. The authors also introduce SeFoRA-Ho, a rank-homogeneous variant, and provide convergence proofs for this setting.\n\n## Findings\nNumerical experiments on fine-tuning RoBERTa-Large on GLUE benchmarks demonstrate that SeFoRA and SeFoRA-Ho outperform existing state-of-the-art methods in both rank-homogeneous and rank-heterogeneous settings. The method successfully handles varying client ranks while maintaining accuracy comparable to or better than methods that operate in the full model space, but with significantly lower server-side computational requirements. The authors also note that by enabling linear aggregation, SeFoRA provides a foundation for future integration with secure aggregation and differential privacy protocols.\n\n## Why It Matters\nSeFoRA provides a scalable and mathematically sound way to perform federated fine-tuning of large foundation models. By decoupling the local client rank from the global aggregation process, it allows for more flexible deployment in real-world federated environments where clients have varying hardware capabilities and data complexities.\n\n## Key Terms and Definitions\n- LoRA (Low-Rank Adaptation) — A parameter-efficient fine-tuning technique that approximates weight updates as the product of two low-rank matrices.\n- Bilinear Mismatch — The mathematical discrepancy that occurs when averaging the factors of a low-rank update ($B_{avg}A_{avg}$) instead of averaging the full updates ($\sum p_i B_i A_i$).\n- Sketching — A dimensionality reduction technique that projects a large matrix into a smaller subspace using random matrices to preserve essential information.\n- Rank Heterogeneity — A setting where participating clients use different adapter ranks, resulting in matrices of incompatible dimensions that cannot be averaged directly.\n- Unsketching — The process of reconstructing a low-rank approximation of the original matrix from its compressed sketch matrices." }