LLM-as-a-Judge for Reliable and Explainable Offline Evaluation in Top-K Recommendation

Research Problem

Traditional offline evaluation of recommender systems relies on Top-K metrics computed over holdout feedback. This approach suffers from two major flaws: it assumes observed feedback is a perfect proxy for user preference (ignoring biases like incomplete item exposure) and it treats the evaluation as a black-box numerical score, offering no insight into why a recommendation was deemed relevant or irrelevant.

The Proposed Framework

The authors propose an LLM-as-a-Judge framework to move beyond rigid ID matching. Instead of comparing item IDs, the framework uses a semantic proxy derived from user textual history to represent true preferences. The evaluation process follows a two-step 'reasoning-then-scoring' paradigm:

1. Reasoning: The LLM analyzes user history to infer latent interests and evaluates the relevance of a candidate item based on its textual features.
2. Scoring: The LLM assigns a binary relevance score (0 or 1) and provides a natural language rationale for its decision.

This approach allows for flexible matching in semantic space, enabling the system to recognize relevant items that might not have been explicitly interacted with in the training data.

By providing explicit rationales for every 'hit' or 'miss,' this framework transforms evaluation from a purely quantitative exercise into a transparent, interpretable process. Experiments on real-world datasets (Kuairec and Coat) demonstrate that this LLM-based evaluation is significantly more consistent with unbiased ground-truth rankings than traditional metrics, providing a more robust way to guide the development and deployment of recommender systems.