Learning Engagement Assistant (LEA): Cross-Course Scalability and Classroom Evaluation of an Agentic AI Tutoring System

This paper evaluates the Learning Engagement Assistant (LEA), an agentic AI tutoring system that combines Retrieval-Augmented Generation (RAG) with structured Knowledge Component (KC) models. While previous research validated LEA through simulations, this study provides the first empirical evidence from a real-world classroom deployment and tests the system's ability to scale across three distinct university courses spanning different academic levels and disciplinary domains.

The authors deployed LEA across three courses: a graduate-level Machine Learning course, an undergraduate Data Structures and Algorithms course, and a graduate-level Human Psychology course. The evaluation consisted of two primary components: a pilot classroom study with eight students to assess usability, trust, and pedagogical perception, and a large-scale technical evaluation using the RAGAS framework to measure the quality of the system's retrieval and generation processes across 660 questions.

The study highlights a significant divergence between simulation-based predictions and actual classroom performance, suggesting that synthetic evaluation alone is insufficient for predicting real-world student interaction. Technically, the system showed high stability in Answer Relevancy and Context Precision across all courses. However, the system's Faithfulness—the accuracy of generated answers relative to the provided context—decreased as the curriculum moved further from the subject matter of the course for which the system was originally designed. This indicates that while the orchestration layer is successfully course-agnostic, the downstream generation components may require further tuning to achieve full cross-disciplinary performance.

As AI-driven tutoring systems are rapidly deployed in educational settings, there is a growing gap between the speed of implementation and the availability of rigorous, evidence-based evaluations. This research provides a necessary framework for assessing whether adaptive tutoring architectures can truly generalize across domains, moving the field beyond simple prompt-response interfaces toward more robust, pedagogically grounded AI agents.