A Self-Evolving Default Action for Cooperative Tasks with Continuous Action Space

Multi-agent reinforcement learning (MARL) often struggles with credit assignment in continuous action spaces. Traditional methods like COMA are restricted to discrete actions, while others rely on restrictive monotonicity assumptions or biased Monte Carlo approximations. This paper asks: How can we design a counterfactual baseline for continuous action spaces that accurately quantifies individual agent contributions while maintaining unbiased policy gradients and convergence guarantees?

The SAFE Approach

The authors propose SAFE (Self-evolving default Action From Experiences). Instead of using complex reward models or simulations to estimate counterfactuals, SAFE uses a centralized critic to evaluate the global action-value function. For each agent, it derives a baseline by replacing the agent's action with a default action sampled directly from that agent's own experience replay buffer. As training progresses, this sampled action naturally evolves to reflect the agent's increasingly optimized behavior. This design allows the baseline to accurately represent the agent's average performance without requiring task-specific prior knowledge or additional environment simulations.

Mathematical analysis proves that the SAFE baseline introduces zero bias into the deterministic policy gradient, ensuring convergence to local optima. Empirical evaluations on cooperative autonomous driving tasks—ranging from 2 to 7 vehicles—show that SAFE consistently outperforms state-of-the-art baselines such as VDN, QMIX, and COMA. Ablation studies confirm that these performance gains are specifically attributable to the credit assignment mechanism rather than just the adaptation to continuous action spaces. Furthermore, the authors demonstrate that using a single sampled default action is more effective than averaging multiple samples, validating the efficiency of their design.