Direct Advantage Estimation for Scalable and Sample-efficient Deep Reinforcement Learning

Deep reinforcement learning (RL) often requires massive amounts of data to train agents, and many existing methods struggle with partial observability—where the agent cannot see the full state of the environment. While Direct Advantage Estimation (DAE) has shown promise in improving sample efficiency by learning the advantage function directly, it was previously limited to fully observable environments and suffered from high computational overhead due to the need to model complex transition probabilities. This paper addresses these limitations to make DAE more applicable to realistic, partially observable settings.

The authors make two primary contributions. First, they generalize the theoretical framework of DAE to Partially Observable Markov Decision Processes (POMDPs). By replacing state-based inputs with histories (information vectors), they show that the return can be decomposed into advantage functions even when the full state is hidden. Second, they introduce a discrete latent dynamics model to approximate transition probabilities. Instead of learning expensive generative models for high-dimensional observations, they embed observations into a compact latent space and use a Winner-Takes-All (WTA) loss to capture stochastic transitions efficiently. This allows the agent to perform off-policy corrections without the computational burden of traditional methods.

The researchers evaluated their approach on 47 games from the Arcade Learning Environment. They found that the method scales effectively with the capacity of the function approximator, meaning that larger models lead to better performance. Notably, the proposed DAE variant achieves performance comparable to the Rainbow DQN baseline while using only 10% of the training data. Furthermore, the agent demonstrates competitive performance against modern model-based approaches like DreamerV3, confirming that the method is both sample-efficient and capable of handling complex, partially observable dynamics.