Statistical Inference for Misspecified Contextual Bandits

Adaptive experiments, such as those used in mobile health or recommender systems, often rely on contextual bandit algorithms to balance exploration and exploitation. While these algorithms are effective for maximizing rewards, they introduce complex dependencies that complicate post-experiment statistical inference. This paper investigates how to perform valid inference (e.g., confidence intervals and hypothesis tests) when the underlying outcome model is misspecified—a common scenario in real-world applications where true reward mechanisms are unknown or simplified for computational efficiency.

The authors identify that popular bandit algorithms like LinUCB can become unstable when the working model is misspecified, causing the estimator to exhibit non-Gaussian behavior and invalidating standard inference. To address this, they develop an inverse-probability-weighted Z-estimation (IPW-Z) framework. This framework targets a broad class of parameters, including projection parameters and off-policy values, without requiring a correctly specified outcome model. The authors introduce a stability condition called "scaled inverse-propensity convergence," which ensures that the IPW-Z estimator remains consistent and asymptotically normal even when action-specific sampling rates vary over time.

The study establishes that policy stability is a prerequisite for valid inference in adaptive settings. By analyzing the properties of various policy classes, the authors show that policies which avoid tightly coupling exploration to misspecified reward models are more stable. Specifically, they find that smooth allocation rules (e.g., Boltzmann sampling with sufficient temperature) are more robust than sharp, argmax-based policies like LinUCB. The proposed IPW-Z framework provides a reliable way to conduct inference, as demonstrated through simulations and a real-data-calibrated application using HeartSteps V1, where the method achieves reliable coverage for confidence intervals.