Federated Lightweight Fine-Tuning

Federated learning is typically bottlenecked by the need to transmit full model weights or large gradient updates between clients and a central server. FLITE (Federated Low-rank Iterative Training Engine) addresses this by using mapping networks. Instead of communicating weights, clients train a small, low-dimensional latent vector. This latent is passed through a shared, frozen, low-rank affine projection to generate the actual weight updates. Because the projection is affine and shared, averaging the latents on the server is mathematically equivalent to averaging the full weight updates, preserving standard FedAvg semantics while drastically reducing the payload.

To make this approach practical for large-scale models, the authors introduce three key design choices:

• Delta Formulation: Rather than generating full weights, the latent generates an additive correction (delta) around a shared, centrally-pretrained base model. This makes the optimization task significantly easier and more stable.
• Low-Rank Factorization: The projection matrix is factored into low-rank components, reducing the memory footprint of the generator from tens of gigabytes to approximately 10 MB, allowing it to run on standard hardware.
• Frozen Orthogonal Classifier: By freezing the final classification head with an orthogonal initialization, the authors remove the largest layer from the communication payload entirely while simultaneously improving model accuracy.

FLITE achieves an 8,718x reduction in communication bandwidth on CIFAR-100, transmitting only ~5 KB per client per round compared to ~45 MB for standard FedAvg. Crucially, it maintains accuracy within 0.5 percentage points of the full-weight baseline. Furthermore, the low-dimensional nature of the latent makes it highly robust to quantization; using int4 precision reduces the payload to just 648 bytes per round without any loss in accuracy, whereas full-weight FedAvg collapses under similar quantization. This makes FLITE highly effective for resource-constrained edge devices.