To Tab or Not to Tab: Measuring Critical Engagement in AI Code Completion Tools Using Behavioral Signals and Attention Checks

As AI code completion tools like GitHub Copilot become standard in computing education, concerns have emerged regarding whether students are critically evaluating the code they accept or merely relying on the tool to bypass cognitive effort. This paper introduces Clover, a custom IDE extension designed to measure student interaction patterns with AI suggestions. By logging fine-grained behavioral data and injecting deliberate, semantically incorrect 'attention checks' into the coding workflow, the authors sought to quantify the relationship between interaction styles, reflective engagement, and task performance.

To move beyond qualitative observations, the researchers developed a taxonomy of interaction metrics, including tab-accepts, slow-accepts, and post-acceptance modifications. The study involved 55 students in a CS1 course who completed a programming task using the Clover tool. During the task, the system intermittently provided misleading code suggestions. If a student accepted one of these suggestions without modification, it was recorded as a failed attention check, serving as a proxy for unreflective engagement.

Analysis of the interaction logs revealed that student engagement with AI suggestions is highly heterogeneous. While no single metric was a strong predictor of overall task performance, the study found a clear, strong correlation between high rates of 'tab-accept' behavior and failed attention checks. Conversely, longer dwell times—the duration a student spends reviewing a suggestion before acting—were associated with better performance on attention checks. These results suggest that while students use AI tools in diverse ways, rapid, uncritical acceptance is a measurable indicator of reduced cognitive engagement.

This work provides a scalable, quantitative framework for educators to monitor how students interact with AI assistants. By formalizing a taxonomy of interaction behaviors, the authors offer a path toward identifying students who may be over-relying on AI, potentially enabling more targeted pedagogical interventions that encourage critical thinking rather than passive acceptance.