Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth

Evaluating large language models (LLMs) in subjective, culturally sensitive domains like Arabic sociolinguistics is difficult because standard metrics often fail to capture nuance. This paper introduces a cross-evaluation framework to determine if frontier LLMs can effectively serve as judges for Arabic cultural and linguistic tasks. The authors created a dataset of 103 prompt-rubric pairs covering Egyptian and Iraqi Arabic, authored by native-speaker subject matter experts (SMEs).

The framework employs a provider-level self-evaluation guard, ensuring that no model evaluates responses from its own provider. The authors use a dual-metric scheme—Mean Absolute Deviation (MAD) and Signed Mean Error—to distinguish between symmetric grading noise and directional bias (such as leniency). The study evaluates three target models (Gemini, Muse Spark, and Grok) using five different frontier LLMs as judges.

• Judge Reliability: GPT-5.4 emerged as the most reliable judge, exhibiting the lowest MAD and near-zero directional bias. In contrast, four of the five judge models showed systematic leniency, which is particularly problematic on penalty-weighted rubrics where leniency masks the error rates the rubric is designed to detect.
• Domain Difficulty: All judge models performed worse on cultural tasks compared to linguistic tasks. This gap highlights the failure mode of implicit cultural reasoning; while linguistic tasks often involve verifiable morphological or syntactic checks, cultural tasks require the judge to simulate native-speaker judgment, which remains a significant hurdle for current models.
• Target Performance and Gradeability: The study found that target model quality does not scale linearly with how easily a model is graded. Interestingly, moderate-quality models that provide concise answers are often easier to grade than high-quality models that provide verbose, nuanced responses, as the latter create more opportunities for judges to misinterpret or over-penalize minor elaborations.

This research demonstrates that LLM-as-a-judge frameworks cannot be treated as a plug-and-play solution for culturally sensitive domains. The systematic leniency and the difficulty of grading implicit cultural knowledge suggest that automated evaluation requires careful rubric design and model selection. By highlighting that grading consistency is tied to the nature of the task (cultural vs. linguistic) rather than just model capability, the authors provide a roadmap for more robust, domain-specific AI evaluation.