Constrained Reinforcement Learning Using Successor Representations

In real-world reinforcement learning (RL), agents must often adhere to safety constraints (e.g., avoiding obstacles or staying within power limits). Standard RL methods struggle when these constraints change over time, as they typically require full retraining of the policy network. This paper investigates how to make constrained RL agents more flexible by decoupling the environment's dynamics from the specific reward and cost functions.

The authors introduce the Safe Deep Successor Representation (SafeDSR), which extends the existing Deep Successor Representation (DSR) framework to Constrained Markov Decision Processes (CMDPs). The core innovation is to learn a set of successor features that capture the environment's dynamics independently of the reward or cost signals. By doing so, the agent learns a representation of state reachability that remains valid even if the cost function changes. When the cost structure shifts, the agent only needs to update a lightweight weight matrix—a supervised learning task—rather than retraining the entire deep neural network.

The authors demonstrate that SafeDSR is significantly more flexible than traditional baseline methods like SafeDQN. In a custom two-dimensional navigation environment, SafeDSR maintains performance competitive with standard algorithms while enabling rapid adaptation to new cost distributions within a few environment interactions. The study also introduces new evaluation metrics, specifically the 'Safe Goal Count' and 'Average Safe Goal Rate,' which provide more stable and interpretable performance measures than traditional episodic reward averages.

This research addresses a critical bottleneck in deploying RL agents in dynamic real-world environments. By allowing agents to adapt to new safety requirements without expensive retraining, SafeDSR makes it feasible to deploy autonomous systems in settings where constraints are not static, such as robotics, power-grid management, or workforce scheduling.