Explainable Reinforcement Learning via Physics-Aware Policy Distillation

As deep reinforcement learning (DRL) systems are increasingly deployed in safety-critical sectors like robotics and automotive engineering, their "black-box" nature poses significant risks for regulatory compliance and human trust. This paper investigates whether a high-performance continuous control DRL agent can be distilled into a transparent, interpretable surrogate model without sacrificing performance or stability.

The researchers utilize the classic Inverted Pendulum benchmark to test their framework. They first train a Twin Delayed DDPG (TD3) agent as an opaque "teacher" model. This policy is then distilled into a shallow Decision Tree "student" model. To ensure the student learns robust control logic, the authors employ "Noisy Oracle Rollouts"—injecting noise into the expert's actions to force the agent into near-failure states, thereby capturing critical recovery behaviors. Furthermore, they introduce a custom "Pole Urgency" feature, which combines angular position and velocity to help the decision tree learn non-linear, diagonal decision boundaries that standard axis-aligned splits would otherwise miss.

The distilled decision tree successfully matches the performance of the expert TD3 teacher, achieving a 100% success rate in maintaining the pendulum's balance over 1,000-step horizons. The study confirms that the surrogate model provides both global interpretability (via a 2D decision boundary map) and local interpretability (via real-time decision path tracking). However, the analysis reveals a fundamental trade-off: the quantization of continuous actions into discrete leaf nodes induces "Bang-Bang" control behavior, resulting in a stable bimodal limit cycle where the system oscillates around the equilibrium point rather than remaining perfectly static.