How AI Assistance Affects Human Skill Development: A Study of Learning with Logic Puzzles

This study investigates the tension between short-term performance gains and long-term skill development when humans use AI assistance. While AI tools can help users solve problems immediately, they may also reduce the cognitive effort required to learn underlying skills. To examine this, the researchers conducted a controlled experiment using a logic-puzzle task across three distinct phases: an initial AI-free baseline assessment (Phase 1), an intermediate phase with on-demand AI assistance under experimentally varied cost conditions (Phase 2), and a final AI-free assessment (Phase 3). The study analyzed how the cost of AI requests influences usage frequency and how this engagement relates to subsequent unassisted performance. Additionally, a Bayesian latent ability model was fitted to separate initial ability, post-AI ability, and participant-specific skill change.

The results show that participants in the lower-cost AI condition requested assistance much more frequently than those in the high-cost condition. Importantly, participants who used AI assistance during Phase 2 performed worse after assistance was removed in Phase 3 compared to those who did not use AI. Furthermore, relying on AI during Phase 2 led to a systematic overestimation of subsequent unassisted performance when predicted from assisted performance. However, the Bayesian modeling revealed that skill development was not simply a function of AI request frequency; rather, greater independent problem-solving effort during Phase 2 was strongly associated with larger gains in latent ability, suggesting that learning is impaired only when AI bypasses active reasoning.

As AI tools become ubiquitous in educational, professional, and everyday cognitive tasks, understanding their impact on human capability is critical. This work demonstrates that short-term success achieved through AI does not guarantee durable skill acquisition. The findings highlight the importance of designing AI systems that complement human reasoning and preserve opportunities for independent problem-solving rather than automating away the cognitive processes necessary for learning.