Playing Atari with Deep Reinforcement Learning

How can deep neural networks be effectively integrated with reinforcement learning (RL) to enable agents to learn control policies directly from high-dimensional sensory data, such as raw video, without relying on hand-crafted features or domain-specific knowledge?

The researchers developed the Deep Q-Network (DQN), a convolutional neural network that approximates the optimal action-value function. To address the inherent instability of combining non-linear function approximation with RL, they employed two key techniques:

1. Experience Replay: Storing the agent's experiences in a memory buffer and sampling them randomly for training, which breaks the correlation between consecutive samples and stabilizes the learning distribution.
2. Target Network Updates: Periodically updating the target Q-values to prevent the feedback loops that often cause divergence in off-policy learning.

The model was tested on seven Atari 2600 games using the Arcade Learning Environment, with a single, fixed architecture and set of hyperparameters applied across all games.

The DQN demonstrated remarkable versatility and performance. It outperformed all previous RL methods on six of the seven games and exceeded human-level performance on three (Breakout, Enduro, and Pong). The study confirms that deep neural networks can successfully extract relevant visual features for control tasks without prior knowledge of the environment's internal state or object definitions. Furthermore, the use of experience replay and stochastic gradient descent allowed for stable training, avoiding the catastrophic divergence often associated with non-linear function approximation in RL.

This paper represents a landmark in artificial intelligence, proving that a single, general-purpose architecture can learn to master diverse, complex tasks directly from raw sensory input. By bridging the gap between deep learning and reinforcement learning, it laid the foundation for modern deep RL, demonstrating that agents can learn to perceive and act in environments that were previously considered too complex for automated control.