EVALUATING INTERACTIVITY: TOWARD AUTOMATED ASSESSMENT OF AI-GENERATED EXPLORABLE EXPLANATIONS

As generative AI becomes increasingly capable of producing interactive learning materials—such as explorable visualizations and simulations—the field faces a significant evaluation gap. Current benchmarks primarily focus on static outputs, such as visual fidelity, code executability, or textual correctness. These metrics fail to capture the dynamic, learner-controlled nature of explorable explanations, where the pedagogical value depends on how a user interacts with the system to reach different states and receive feedback. Because this interaction logic is often buried in complex, low-level code, it has remained difficult to evaluate systematically.

To address this, the authors introduce EE-Eval, a framework that externalizes implicit interaction logic by representing explorable explanations as Finite State Machines (FSMs). By mapping an interface into a graph of states, user-triggered transitions, and system responses, EE-Eval transforms interaction into a machine-interpretable structure. The framework uses a few-shot prompting strategy to extract these FSMs from AI-generated artifacts. Once extracted, the generated FSMs are compared against ideal, pedagogically grounded reference FSMs using a combination of structural graph metrics and semantic embedding comparisons. This approach allows for a multi-dimensional assessment of whether an AI-generated tool provides the intended exploratory experience.

EE-Eval shifts the paradigm of evaluation from passive output verification to active behavioral modeling. By framing interactivity as a testable, structured artifact, the framework provides researchers and educators with a diagnostic tool that aligns closely with human judgments of pedagogical effectiveness. In empirical tests across thousands of generated explanations, EE-Eval consistently outperformed existing vision-based and unit-testing baselines, demonstrating that it can successfully distinguish between high-quality, coherent interactions and those that are merely visually appealing or functionally correct. This work provides a scalable path toward more reliable, pedagogically sound AI-assisted educational content.