Evaluating Japanese Dialect Robustness Across Speech and Text-based Large Language Models

As Large Language Models (LLMs) are increasingly integrated into speech-processing pipelines to create Speech Language Models (SLMs), it remains unclear how well these systems handle linguistic diversity. Specifically, this paper investigates whether the dialectal comprehension capabilities of a base LLM are preserved when it is adapted for speech, and whether specific training interventions can improve performance on regional Japanese dialects.

The researchers evaluated four LLMs (Llama-3.1, Swallow, LLMJP, and Sarashina) and their corresponding SLM variants across 20 Japanese dialects. They defined "dialectal robustness" as the ratio of performance on dialectal input versus standard input, using a translation task from Japanese to English as the primary benchmark. This metric allows for a fair comparison across models of varying base capabilities. The study systematically tested three interventions: using a base LLM with known dialectal strengths, incorporating dialectal training data (the CPJD corpus), and fine-tuning the speech encoder (Whisper-Large-V3) to better align acoustic features with the LLM.

The experiments revealed a clear performance gap between standard and dialectal inputs across all models. Crucially, the researchers found that an SLM’s ability to handle dialects is highly dependent on the base LLM's inherent robustness; if the text-based model struggles with a dialect, the speech-based version generally struggles as well. However, the study demonstrated that this limitation is not fixed. By fine-tuning the speech encoder and training on dialect-specific datasets, the models showed measurable improvements in robustness. This suggests that while the LLM provides the foundational linguistic knowledge, the acoustic front-end plays a critical role in successfully bridging the gap between dialectal speech and the model's internal representations.