EduPanel: A Three-Agent LLM Judge for Teaching Videos — Reliability, Complementarity, and Human Trust Calibration

EduPanel addresses the challenge of evaluating teaching videos by moving beyond universal quality scores. Instead, it uses a three-agent architecture to provide learner-conditioned assessments. The system decomposes the evaluation process: Agent 1 acts as a content analyst to extract multimodal evidence; Agent 2 focuses on objective, text-based scoring; and Agent 3 simulates a specific learner persona to assess pedagogical fitness. This design aims to provide both a final score and inspectable intermediate evidence, such as content maps and identified issues, to assist human experts.

The researchers evaluated EduPanel against human experts using a two-round, verify-then-revise protocol. In the first round, experts rated videos blindly; in the second, they reviewed EduPanel's feedback before finalizing their scores. The study found that EduPanel’s reliability is comparable to that of a median human expert. Furthermore, the inclusion of EduPanel’s feedback improved the scoring accuracy of human experts, reducing the Mean Absolute Error (MAE) from 0.87 to 0.73.

A critical aspect of the study was testing whether experts would blindly accept AI feedback. By planting deliberate errors in the AI's output, the researchers found that experts were able to distinguish between reliable and unreliable AI assessments (AUC = 0.77). This suggests that EduPanel functions effectively as a collaborative assistant rather than a replacement, as experts can critically verify the AI's rationale and reject incorrect suggestions.