Is Solving Better Than Evaluating GenAI Solutions?

This study utilized a randomized A/B crossover design with 220 students in a junior-level algorithms course to compare two pedagogical approaches: traditional problem-solving versus evaluating GenAI-generated solutions. Over six assignments, student groups alternated between these two conditions. The evaluation-centered tasks required students to prompt an LLM for a solution to a challenging algorithmic problem and then identify, explain, and grade the errors within that output. The researchers measured the impact of these interventions through midterm and final exam scores, performance on exam problems structurally aligned with the homework, and student survey data regarding study habits and perceived helpfulness.

The study found no statistically significant difference in summative performance—such as midterm or final exam scores—between students who solved problems directly and those who evaluated GenAI-generated solutions. Although students in the evaluation condition achieved higher scores on the homework assignments themselves, this did not lead to improved conceptual transfer or better performance on related exam questions. Survey results indicated that most students did not change their study habits, though those who did adapt their strategies reported that the evaluation tasks were more helpful.

As GenAI becomes ubiquitous, educators are increasingly shifting from traditional problem-solving to tasks that emphasize critique and verification. This research provides critical empirical evidence that simply replacing creation-based tasks with evaluation-based tasks is not a silver bullet for learning. The findings suggest that while evaluation tasks can be integrated into a curriculum without harming student performance, they do not automatically produce deeper conceptual understanding. To achieve meaningful learning gains, educators may need to provide more deliberate scaffolding that forces students to move beyond surface-level error diagnosis and into deeper algorithmic reasoning.