State Representation Matters in Deep Reinforcement Learning: Application to Energy Trading

Research Question and Approach

This paper investigates how different state representations affect the performance and generalization of deep reinforcement learning (RL) agents in energy storage arbitrage. The authors use a fixed Double DQN (DDQN) agent and a controlled environment (HydroDam) to isolate the impact of input features. By keeping the environment, reward function, and network architecture constant, they compare three families of features: absolute (price levels and calendar time), relative (price comparisons against recent history), and forecast (short-horizon price predictions).

To test generalization, the researchers train and select policies using 2007–2011 Belgian day-ahead prices. They then evaluate these policies on two challenging test settings: a later, more volatile period in the same market (2012–2025) and 39 unseen European market zones. This setup reveals whether a policy is merely overfitting to the training regime or if it can adapt to shifting price levels, volatility, and market structures.

The study finds that state representation is a critical design choice rather than a minor preprocessing step. Policies relying solely on absolute features perform well during validation but fail to generalize, dropping to 28.8% of optimal profit in the test set. Conversely, relative or forecast features alone provide insufficient information about the scale of trading opportunities. The most robust results are achieved by combining all three feature families, with the 'absolute + relative + forecast' model reaching 55.6% of optimal profit on the test set and 47.5% across diverse market zones, significantly outperforming single-feature models and a strong heuristic baseline.

Energy markets are inherently non-stationary, characterized by shifting price regimes and extreme events like the 2021–2022 energy crisis. This research demonstrates that for RL agents to be effective in real-world energy trading, they must be provided with a multi-faceted view of the market. By combining absolute, relative, and forecast information, agents can better navigate both the current price context and the broader market trends, ensuring more reliable performance when market conditions deviate from the training data.