Shahin Hossain, Tukhbita Afroz Nawmi
5 min
As generative AI (GenAI) becomes ubiquitous in higher education, the binary question of whether students use AI has become insufficient for understanding its impact on learning and academic integrity. This study sought to move beyond simple usage metrics by developing a theoretically grounded instrument—the GenAI Reliance Types Scale (GenAI-RTS)—to measure how students rely on AI during the extended, iterative process of academic writing.
The researchers employed a deductive, theory-driven approach to define four primary types of reliance: Strategic, Instrumental, Dependent, and Dialogic. They developed a 20-item scale and validated it using a multi-source framework, including a survey of 382 undergraduates at a U.S. Minority-Serving Institution and qualitative interviews with 14 students. The validation process included confirmatory factor analysis (CFA) to test competing models, Rasch analysis to refine the response format, and tests for scalar measurement invariance across gender, first-generation status, and academic discipline.
The study confirmed a five-factor structure for the GenAI-RTS. The initial concept of 'Strategic Reliance' was empirically split into two distinct facets: Deliberate Use and Critical Evaluation. These, along with Instrumental, Dependent, and Dialogic reliance, form a robust framework for profiling student behavior. The scale demonstrated acceptable to good reliability and, notably, achieved scalar measurement invariance across key demographic groups, suggesting that the instrument can be used to make fair comparisons between different student populations. Furthermore, the study found that Strategic reliance is positively correlated with AI literacy, and the different reliance profiles effectively differentiate students based on their writing processes and outcomes.
This instrument provides educators and researchers with a standardized tool to move past the 'use vs. non-use' debate. By identifying specific reliance profiles, institutions can better design targeted AI literacy interventions, assess the impact of AI on cognitive development, and address potential equity gaps in how different student groups engage with these technologies.
As generative AI (GenAI) becomes increasingly embedded in undergraduate academic writing, how students rely on these tools, rather than simply whether they use them, has become a central question for learning, academic integrity, and educational equity. Existing measures of reliance were developed inductively, focused on discrete problem-solving tasks, and validated mainly with homogeneous samples. This study developed and validated the GenAI Reliance Types Scale (GenAI-RTS), a 20-item instrument measuring four theoretically derived types of GenAI reliance: Strategic, Instrumental, Dependent, and Dialogic. Validation followed the multisource framework of the Standards for Educational and Psychological Testing, drawing on a survey of 382 undergraduates at a U.S. Minority-Serving Institution and interviews with 14 purposively sampled students. Confirmatory factor analyses of six competing models supported a five-factor structure in which Strategic Reliance comprises two facets, Deliberate Use and Critical Evaluation, alongside Instrumental, Dependent, and Dialogic factors (CFI = .92, RMSEA = .08; DWLS CFI = .98, RMSEA = .07). Subscale reliability was acceptable to good (omega = .75-.88), and scalar measurement invariance held across gender, first-generation status, and STEM/non-STEM majors, to our knowledge the first such evidence for a GenAI reliance instrument. Rasch analysis indicated that a five-point response format would improve category functioning. Strategic reliance was positively associated with AI literacy, and the reliance types differentiated students across multiple writing process and outcome variables. The GenAI-RTS offers researchers and educators a theoretically grounded, psychometrically validated instrument for identifying undergraduate reliance profiles and supporting research, assessment, and AI literacy intervention.
Alex: That's exactly the right question. They compared six different mathematical structures against their student data to see which one fit best. And the winner required splitting what they'd initially called "Strategic" use into two separate parts: planning and evaluating.
Sam: Why split them?
Alex: Because the data showed they're genuinely different skills. Students often evaluate AI output—checking whether what it produced is actually good—far more than they use AI to plan their work in the first place. Forcing those two behaviours into one category made the model less accurate.
Sam: That's a bit like assuming a chef who's great at tasting and adjusting a sauce must also be great at planning the menu. They're related skills, but not the same one.
Alex: That's a useful way to put it. They also ran something called Rasch Analysis on the individual survey questions—think of it as checking that each question in the survey is pulling its weight fairly, like making sure every hurdle in a race is set to the right height.
Sam: So they're not just testing students—they're testing the test itself.
Alex: Precisely. And they took one more step: they checked whether the survey measured the same thing consistently across different groups of students—different genders, different academic backgrounds, different majors. That process is called scalar invariance testing, and it's essentially a fairness check.
Sam: That matters a lot. A tool that works well for one group of students but gives skewed results for another isn't really useful for educators trying to make fair judgements.
Alex: Exactly. Without that check, you might end up with a scale that looks reliable on average but quietly misrepresents certain students. By confirming it holds up across groups, the researchers give educators something they can actually trust.
Sam: So the shift this paper is proposing is from a yes-or-no question—"did you use AI?"—to something much more informative: "how are you relying on it, and what does that tell us about your learning?"
Alex: That's the core of it. It gives educators a shared language—a way to distinguish between a student who is genuinely engaging with AI as a thinking tool and one who has stepped back from the thinking altogether. And that distinction, the paper argues, is where the real educational conversation needs to happen.
Sam: It also reframes the integrity question. Rather than treating AI use as inherently suspicious, it opens up the possibility of actually supporting students who are struggling—before a problem becomes a crisis.
Alex: That's well put. The goal isn't surveillance. It's understanding. And that's a meaningfully different starting point for both educators and students.
Sam: Thanks for walking us through this one. It's a good reminder that the tools change faster than our frameworks for understanding them—and that building better frameworks is genuinely useful work.
Alex: It is. Thanks for listening to ResearchPod.