A Distribution Mapping Approach to Counterfactually Fair Reinforcement Learning

Reinforcement learning (RL) agents deployed in high-stakes environments, such as healthcare, often learn policies that inadvertently discriminate against specific subpopulations. This paper addresses the challenge of enforcing counterfactual fairness (CF) in sequential decision-making, where the goal is to ensure that an agent's decisions for an individual would remain unchanged if their sensitive attribute (e.g., race or gender) were different, while holding all other causal factors constant.

The authors propose a two-stage framework called Counterfactually Fair Sequential Marginal Distribution Mapping (CFSMDM). First, they use a novel quantile distribution mapping technique to preprocess training data. This step estimates counterfactual states and rewards by identifying the quantile level of observed variables and mapping them to the corresponding quantile of the distribution under a counterfactual sensitive attribute. This approach is more flexible than existing methods because it does not require the noise in state and reward transitions to be additive. Second, the agent performs standard offline policy learning on these augmented, "de-biased" trajectories.

The paper provides theoretical bounds for both the suboptimality gap (the loss in performance compared to an optimal policy) and the level of counterfactual unfairness for policies learned via CFSMDM. The authors demonstrate that under mild regularity conditions, the algorithm effectively removes sensitive information from the training data, allowing for the construction of policies that satisfy counterfactual fairness. Empirical results on both numerical simulations and a real-world digital health dataset confirm that the proposed method successfully mitigates unfairness while maintaining competitive decision-making performance.