ROBUST CONTROL UNDER STATIONARY AMBIGUITY

When training control policies in simulators, researchers often randomize parameters to account for uncertainty. A common method is static randomization, where a parameter is drawn once at the start of a trajectory and held constant. While this forces the policy to be robust initially, it creates a fundamental problem: as the policy observes the system over time, it gradually infers the true parameter value. This leads to vanishing ambiguity, where the policy becomes increasingly specialized to its estimate of the latent state. In environments like financial markets, where latent factors (such as volatility regimes) shift, this specialization is a liability; the policy loses its robustness exactly when it needs it most.

To address this, the authors introduce the concept of stationary ambiguity. Instead of forcing the policy to learn a fixed parameter, the simulator is designed so that the ambiguity about the latent state remains constant over time. This is achieved by ensuring the simulator induces a stationary filter process. In practice, this means the latent state is modeled as a dynamic process rather than a static variable, preventing the policy from ever fully 'solving' for the parameter. By providing a warm-up period of observations, the simulator ensures that the policy enters the control phase with a stable, non-vanishing level of uncertainty.

This research provides a rigorous framework for designing simulators that produce more resilient control policies. By shifting the focus from static parameter uncertainty to stationary filter dynamics, the authors show that policies can maintain robustness to regime changes in real-world applications like derivative hedging. This approach offers a practical recipe for simulator design, suggesting that how we initialize and randomize latent variables is just as important as the model architecture itself for ensuring long-term performance in stochastic control tasks.