PFAdapter: Hierarchical LoRA Decomposition for Personalized Federated MLLMs

As Multimodal Large Language Models (MLLMs) become the cognitive engines for edge-based agentic AI, they face a critical trade-off in federated learning: how to aggregate global knowledge without diluting the local, task-specific expertise required for heterogeneous edge environments. Existing federated fine-tuning methods often use uniform parameter aggregation, which leads to 'weight washing'—where local nuances are erased—and high communication overhead due to the transmission of all model parameters.

The authors propose PFAdapter, a framework that introduces hierarchical LoRA decomposition. Instead of treating all adapter parameters as a single block, PFAdapter categorizes them based on their functional role within the self-attention mechanism. Query and key projections are designated as global-shared components, as they capture universal multimodal semantics. Conversely, value and output projections are designated as local-private components, as they are responsible for edge-specific adaptation. To ensure these components remain distinct, the authors employ an orthogonality regularization technique based on the Frobenius norm, which prevents redundant feature learning between the global and local subspaces.

PFAdapter enables a selective aggregation protocol where only the global-shared components are synchronized across the federated network. This approach reduces communication traffic by nearly 50% compared to standard federated LoRA methods. Experimental results across diverse benchmarks—including medical VQA (VQA-RAD, SLAKE) and social multimodal datasets (Hateful Memes, CrisisMMD)—demonstrate that PFAdapter consistently outperforms state-of-the-art baselines, achieving accuracy improvements ranging from 2.4% to 4.8%. By preserving local expertise while sharing universal multimodal representations, the framework provides a more robust and efficient path for deploying agentic AI in resource-constrained edge networks.