Counterfactual Learning of New Adaptive Instructional Policies using Logged Data

Optimizing Intelligent Tutoring Systems (ITS) traditionally requires either live experimentation, which can be disruptive or unethical, or simulations, which often fail to capture real-world student behavior. This paper proposes a model-free, offline contextual bandit framework that learns adaptive instructional policies directly from existing interaction logs. By mapping students and items onto a continuous latent proficiency-difficulty scale using a Rasch model, the authors treat the tutoring process as a continuous bandit problem. This allows the system to recommend optimal exercise difficulties based on a student's estimated proficiency.

The authors define a reward function inspired by the concept of flow, which incentivizes the system to assign the most difficult task a student can successfully complete. This reward is computed directly from observed outcomes, avoiding the need for complex long-term learning metrics. To evaluate and improve policies, the authors estimate round-specific behavior policies from the logs. These serve as propensity models for off-policy evaluation, enabling the use of estimators like Inverse Propensity Scoring (IPS) and Doubly Robust (DR) methods to learn policies that outperform the original logged behavior.

This framework provides a scalable, data-driven path for improving adaptive learning systems. Because it learns from historical data and avoids the need for a student simulator, it reduces the risk of overfitting to synthetic models. Furthermore, the ability to visualize the learned policies on a continuous scale offers practitioners a transparent diagnostic tool to understand how difficulty is allocated across different student proficiency levels and time steps.