Revisiting Action Factorization for Complex Action Spaces

Research Question and Approach

This paper investigates how to best factorize complex, hybrid discrete-continuous action spaces in reinforcement learning. While many frameworks support these spaces, there is no consensus on the optimal way to decompose actions for different algorithm families (DQN, SAC, PPO). The authors conduct a comprehensive cross-sectional study across 220 configurations, testing various factorization methods—including independent networks, shared encoders, value decomposition (VDN), and auto-regressive models—across three action space types (discretized, hybrid, continuous).

Methodology

To facilitate this study, the authors introduce two new C++ parallel environments, CoopPush and Hybrid-Shoot, designed to isolate state-dependent inter-action dependencies. They also propose two novel variants, VDN-PPO and PPO-MIX, which utilize a branching critic to improve credit assignment in multi-headed PPO architectures. By standardizing the evaluation across different algorithm families, the authors aim to provide a practical guide for researchers selecting architectures for complex control tasks.

Key Findings

• Shared Encoder Architectures: These provide the best compute-to-performance trade-off for most settings.
• Credit Assignment: VDN-PPO and PPO-MIX significantly outperform standard shared-encoder PPO in discrete action spaces by redistributing credit to action heads with higher state agency.
• Mixing Networks: Monotonic mixing (as seen in QPLEX) does not offer performance improvements over shared encoders in simple single-agent settings.
• Action Type: The choice of factorization strategy has a greater impact on performance than the underlying action type (discrete vs. continuous vs. hybrid).