ORCAID: Oblique Rule-Based Continuous-Action Interpretation for Deep RL Policies

Deep reinforcement learning (RL) has achieved remarkable success in complex control tasks, but the resulting neural network policies are often opaque, making them difficult to verify or trust in safety-critical applications. The authors address the challenge of creating human-interpretable surrogate models for agents operating in mixed continuous-discrete state spaces with continuous action spaces.

ORCAID (Oblique Rule-Based Continuous-Action Interpretation) extracts interpretable policies through a three-stage process:

1. Data Collection: The agent interacts with the environment to collect trajectories. To ensure robust coverage, the authors use k-means clustering on visited states and sample from these centroids with added Gaussian noise to mitigate overfitting.
2. Tree Learning: The method constructs an oblique decision tree where each split is a hyperplane (a linear combination of features). This allows for more compact state-space partitioning compared to traditional axis-aligned trees. Each leaf node fits a local linear regression model to predict continuous actions.
3. Simplification: The model reduces complexity by merging adjacent leaf regions and performing backward feature elimination, resulting in a concise set of IF-THEN rules that map state conditions to linear action models.

Experiments across nine control tasks demonstrate that ORCAID produces surrogate models that maintain high fidelity to the original deep RL policy while significantly reducing model size. The resulting rules are not only more interpretable than standard decision trees but can also be used to identify actionable insights that improve the performance of the original agent. The oblique splitting strategy allows the model to capture complex, non-orthogonal decision boundaries with fewer parameters than axis-aligned alternatives.