Multi-AUV Ad-hoc network-based Target Tracking: A Value Gradient Guidance Multi-Agent Diffusion Reinforcement Learning Approach

Networked autonomous underwater vehicles must cooperatively track maneuvering targets under severe constraints such as low-bandwidth acoustic communication, dynamic network topologies, and uncertain ocean disturbances. Conventional multi-agent reinforcement learning methods often suffer from high-dimensional joint state-action modeling and noise-sensitive policy generation, leading to unstable training and degraded tracking performance in dynamic marine environments.

The authors propose a diffusion-based hierarchical control architecture comprising three closed-loop tiers: a global intelligent control layer, a local online training layer, and a physical action execution layer. This structure enables synergistic optimization across global task allocation, decentralized policy learning, and real-time physical execution feedback within self-organizing ad-hoc networks.

At the core of the local online training layer, the Value-Gradient-Guided Multi-Agent Diffusion Reinforcement Learning algorithm replaces traditional deterministic actors with diffusion policies. It incorporates value gradients into the reverse denoising process to steer generated actions toward higher expected returns. Furthermore, twin value networks with joint optimization and soft target updates mitigate overestimation bias and training oscillations under fluctuating underwater conditions.

By uniting generative diffusion models with actor-critic reinforcement learning and hierarchical control, this framework provides a robust solution for multi-agent coordination under communication constraints. Experimental results demonstrate faster convergence, higher tracking accuracy, and smoother training dynamics compared to conventional baselines.