Priority-Aware Learning-Unlearning Correction for Dynamic Decentralized LoRA Fine-Tuning

As Large Language Models (LLMs) are deployed at the network edge, they are often fine-tuned using decentralized federated learning (DFL) to preserve privacy. However, edge networks are dynamic; devices frequently join or leave due to mobility or energy constraints. This creates a significant challenge: how can a decentralized system continuously adapt the global model to new data while simultaneously removing the influence of departed devices, all without a central server or access to historical training data?

The authors propose a framework based on Orthogonal LoRA (Low-Rank Adaptation). Each device is assigned a unique, frozen orthogonal basis that acts as a contribution coordinate. Because these bases are orthogonal, the model can isolate individual device contributions within the shared adapter space. When a device joins, its knowledge is added to the existing subspace; when a device leaves, its specific projection can be deleted without requiring historical gradient records or full retraining. To manage the limited communication bandwidth of wireless edge networks, the authors analyze the correction process as an optimization problem. They identify that different scenarios—such as high local bias or consensus disagreement—require different corrective actions. They then develop a priority-aware policy that dynamically allocates resources among topology refinement, local correction, proximal damping, and synchronization scheduling based on the dominant error source.

Findings and Significance

The framework successfully enables robust post-event correction in decentralized environments. Experiments demonstrate that the system can effectively incorporate new devices and remove departed ones while maintaining model performance. By transforming complex theoretical error bounds into computable priority proxies (using Fisher information and gradient energy), the system makes intelligent, real-time decisions about how to spend limited communication budgets. This work is significant because it bridges the gap between static decentralized training and the practical, dynamic requirements of edge-based generative AI, providing a scalable way to handle membership churn without the need for a central coordinator.