Adaptive Finite-Budget Training for CVaR Risk-Aware Q-Learning

Risk-aware Q-learning (RaQL) is a powerful tool for optimizing policies under risk-sensitive objectives like Conditional Value-at-Risk (CVaR). However, standard implementations often struggle with finite training budgets because fixed hyperparameters cannot account for the shifting dynamics of the inner risk-estimation loop and the outer Q-value updates. This paper introduces an adaptive training controller that redesigns the training procedure without altering the underlying CVaR objective or the Bellman fixed point.

The controller implements six coordinated mechanisms to stabilize training:

• Per-cell inner-step sizing: Adjusts step sizes based on local per-cell progress rather than a global counter.
• Outer-rate-matched decay: Synchronizes inner-loop decay with the outer Q-learning rate to maintain stability as the Q-table converges.
• Early correction: A temporary correction mechanism for the VaR-like inner variable during the initial 5% of training to prevent early divergence.
• Coverage-first sample allocation: A two-phase strategy that prioritizes state-action coverage early in training and shifts to greedy sampling as the policy matures.
• Progressive suffix aggregation: Uses an expanding window of recent inner-loop targets to reduce variance in the Q-update targets.
• Data-driven calibration: Automatically scales key hyperparameters like learning-rate exponents and inner-loop depth based on observable quantities like the discount factor and available budget.

In financial applications, where downside risk is a primary concern, the reliability of risk-aware models is critical. By replacing fixed-parameter schedules with an adaptive controller, the authors demonstrate an 85% reduction in empirical Bellman residuals compared to baseline methods. In a practical Bitcoin trading experiment, this approach yielded a policy with significantly lower volatility and maximum drawdown compared to a buy-and-hold strategy, proving that adaptive training design can materially improve the performance of risk-aware reinforcement learning in high-stakes environments.