Evaluating RAG Metrics in Applied Contexts: An Experiment, Its Findings and Its Limitations

Evaluating Retrieval-Augmented Generation (RAG) systems is notoriously difficult because standard metrics often fail to capture the nuances of factual accuracy and relevance. This paper presents an empirical study that tests metrics from four popular libraries—Ragas, DeepEval, RAGChecker, and Opik—against human-annotated business data. By comparing these automated scores to human evaluations and traditional recall metrics, the author investigates which automated tools provide the most reliable approximations of human judgment.

The study uses a dataset of 96 question-answer pairs derived from internal business documents. The RAG system retrieves relevant spans and generates answers, which are then scored by two human experts using a 5-point rubric measuring factuality and relevance. The author calculates Pearson correlations between these human scores and the outputs of various automated metrics. The study specifically distinguishes between retrieval metrics (evaluating the quality of retrieved spans) and overall metrics (evaluating the final generated response).

The results reveal that traditional metrics like METEOR correlate surprisingly well with human judgment, while some specialized LLM-based metrics show weaker performance. A significant finding is the high correlation between certain RAGChecker metrics and human scores; however, the author warns that this may be misleading. Because the experiment uses only one RAG system, it is difficult to determine if these metrics are truly measuring system quality or simply reflecting the inherent difficulty of the questions themselves. The author concludes that while correlation analysis is useful for filtering out ineffective metrics, it is insufficient to confirm that a metric is accurately measuring the intended criteria.

As organizations increasingly deploy RAG systems, the ability to reliably monitor performance is critical. This paper highlights that "LLM-as-a-judge" metrics are not a silver bullet. By demonstrating the limitations of simple correlation studies, the author encourages researchers to move toward more robust evaluation methodologies, such as comparing metrics across multiple systems or using pairwise preference rankings, to ensure that evaluation tools are actually measuring system improvements rather than noise.