Beyond Fluency: A Rubric-Based Benchmark for Evaluating Saudi Dialect and Cultural Competence in Large Language Models

As large language models (LLMs) see increased adoption in the Arab world, a critical gap has emerged: standard benchmarks prioritize Modern Standard Arabic (MSA), which is a formal register rarely used in daily life. This study investigates whether models can handle the Saudi dialect, which encodes essential social meaning, nuance, and cultural context. The researchers developed a rubric-based benchmark consisting of 31 expert-authored prompts that require lived, local knowledge rather than textbook facts.

The evaluation methodology is split into two phases to ensure fairness. First, subject-matter experts (SMEs) established a ground truth and decomposed it into atomic, mutually exclusive, and collectively exhaustive (MECE) positive criteria. Second, four state-of-the-art models—Claude Opus 5, Gemini 3.7, GPT-5.6, and Kimi K3—were evaluated against these criteria, with additional penalties applied for any errors (such as hallucinations or register distortions) introduced during generation.

The study reveals that Saudi dialectal competence remains an unsolved challenge. All four models clustered within a narrow performance band (42.7%–53.1%), with no model surpassing 55%. A key finding is that the primary failure mode is not factual hallucination (which accounted for only 11.2% of errors) but rather 'Ambiguous Framing' (37.3%). This indicates that models are often fluent enough to avoid overt falsehoods but fail by distorting the register, flattening pragmatic nuance, or mismanaging the cultural context of an interaction.

Furthermore, the researchers identified distinct 'error signatures' for each model. For example, GPT-5.6 showed the lowest hallucination rate but struggled with formatting and omissions, while Gemini 3.7 was heavily prone to ambiguous framing. This consistency-versus-ceiling trade-off suggests that model selection for real-world deployment should be based on matching a model's specific error profile to the requirements of the use case.

This research demonstrates that MSA-centric evaluation is insufficient for real-world Arabic applications. Because dialectal competence is where user experience is often won or lost, developers cannot rely on aggregate fluency scores. By providing a reproducible, rubric-based framework, this study offers a path for creating more culturally aware AI systems that can navigate the complexities of regional Arabic varieties.