DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation

This paper investigates the 'robustness-generation gap' in large language models (LLMs). While modern LLMs have become increasingly adept at understanding dialectal English, they typically default to standard, US-leaning English during generation. The authors introduce DiaLLM, a framework designed to bridge this gap by systematically comparing different adaptation strategies across three model families (Llama 3.1-8B, Qwen 3-8B, and Gemma 3-4B) and three specific English varieties: Australian, Indian, and Northern British English.

The DiaLLM pipeline consists of three stages: continual pretraining (CPT) on the International Corpus of English (ICE), followed by either implicit or explicit post-training paradigms. The authors compare three alignment methods—Direct Preference Optimization (DPO), Group Relative Policy Optimization (GRPO), and Group Sequence Policy Optimization (GSPO)—using a reward signal based on eWAVE morphosyntactic features. By controlling these variables, the researchers isolate the effects of each component on both general benchmark performance and dialectal generation quality.

The study reveals a clear dissociation between dialectal robustness and generation. Standard benchmarks are primarily shaped by CPT and supervised fine-tuning, while alignment methods have a marginal impact on these scores despite visibly altering the model's output style. A significant 'reward-quality gap' was identified: although explicit, variety-targeted adaptation is preferred by human evaluators, the method that most aggressively optimizes the dialectal reward (GRPO) is often not the preferred choice. Independent linguistic analysis confirms that this aggressive optimization can actually result in fewer recognizable dialectal markers, suggesting that current reward designs are insufficient to capture the nuance of natural dialectal speech.