A Noise-Robust Elicit-to-Optimize Framework for Distortion Riskmetrics via Inverse Reinforcement Learning

In many high-stakes decision-making environments—such as robo-advising or autonomous driving—agents often exhibit complex, non-standard risk preferences that are difficult to articulate. While distortion riskmetrics provide a mathematically unified way to describe these preferences (including tail risks and asymmetric concerns), it remains challenging to both identify an agent's specific risk objective and subsequently optimize a policy that adheres to it. This paper addresses this gap by developing an integrated, noise-robust 'elicit-to-optimize' framework.

The authors propose a two-stage approach. First, they use an adaptive Bayesian inverse reinforcement learning (IRL) method to infer an agent's latent distortion riskmetric from their observed, potentially noisy, binary choices. By using adaptive questioning, the framework efficiently narrows down the agent's preference from a candidate set of distortion functions. Second, they develop a model-free reinforcement learning algorithm to optimize policies under the identified risk objective. This is achieved by extending the Proximal Policy Optimization (PPO) algorithm to include a quantile neural network, which approximates the full conditional cost quantile function, allowing the agent to evaluate and optimize general distortion-riskmetric objectives.

The study establishes that a finite set of distinguishing questions is sufficient to identify an agent's preferred distortion riskmetric within a candidate class. The authors prove that their Bayesian IRL algorithm converges at a specific exponential rate, even when agents act suboptimally or inconsistently. The proposed RL algorithm successfully unifies diverse risk objectives by representing them as integrals of the conditional cost quantile function, enabling effective policy optimization in complex, scenario-dependent financial environments. Numerical experiments confirm that the framework is both accurate in preference elicitation and effective in achieving risk-sensitive performance.