Constraint-Aware Aggregation for Federated Reinforcement Learning in Microgrid Energy Coordination

Federated Reinforcement Learning (FedRL) allows distributed energy resources, such as farm-level batteries, to optimize energy usage without sharing raw, private data. However, standard aggregation methods like FedAvg treat all client updates equally, ignoring whether individual policies contribute to system-level violations of shared grid capacity. This paper investigates whether lightweight, constraint-aware aggregation strategies can improve safety in these distributed settings. The authors propose several server-side aggregation rules that reweight client updates based on local reward summaries and attributed violation estimates, avoiding the need for complex dual-variable exchanges or modifications to local training.

To evaluate their approach, the authors introduce DairyGridEnv, a multi-agent environment where five farms coordinate battery storage under stochastic demand and a shared grid capacity constraint. The environment tests the ability of agents to maximize local rewards while adhering to global power limits. The authors validate their methods using both synthetic demand models and real-world load-driven profiles from Finland and the German FIELD dataset, ensuring the results generalize beyond idealized conditions.

The study demonstrates that the choice of aggregation rule is a critical lever for safety in federated systems. Among the tested methods, the penalty-based aggregation rule ($w_i \propto R_i - \alpha V_i$) consistently provides the most reliable trade-off between reward and safety. It significantly reduces grid violations while maintaining higher rewards compared to FedAvg and other baseline strategies. While a combined reward-violation scheme offers a tunable trade-off via a parameter $\lambda$, it proves less stable across different random seeds. These results highlight that incorporating system-level constraints directly into the aggregation process is essential for deploying FedRL in safety-critical energy infrastructure.