Kareem Elozeiri, Mervat Abassy, Omar Kallas, Fahim Dalvi, Preslav Nakov, Kentaro Inui, Nadir Durrani
5 min
Arabic Large Language Models (LLMs) often struggle with dialectal generation, frequently defaulting to Modern Standard Arabic (MSA) due to the scarcity of dialectal training data. This study investigates whether dialectal features are localized in specific neurons or distributed across the model's activation space, and whether these representations can be leveraged to steer model outputs toward specific Arabic dialects at inference time without requiring parameter updates.
The researchers employ two complementary inference-time interventions:
The study evaluates these methods on two Arabic-centric models (ALLaM-7B and Fanar-1-9B) across four dialect groups, using both automated metrics (ADI2) and LLM-as-a-judge protocols.
The results demonstrate that dialectal information in Arabic LLMs is neither purely local nor purely diffuse. Dialect-specific neurons tend to concentrate in the model's final layers, suggesting that dialectal control is most effective when applied near the generation-facing output layers. While neuron-level steering provides fine-grained control, vector steering generally offers more robust performance by capturing distributed features that individual neurons miss. The study confirms that dialect identity is geometrically separable in the model's activation space, allowing for effective, non-invasive control over the model's linguistic register.
This work provides a principled, interpretability-grounded framework for improving dialectal fidelity in LLMs. By demonstrating that dialects can be steered like languages, the authors offer a practical solution for developers to adapt models to specific regional varieties without the high cost and data requirements of full fine-tuning. This approach also deepens our understanding of how multilingual and multidialectal knowledge is organized within transformer architectures.
A key challenge in Arabic NLP is the scarcity of dialectal data relative to Modern Standard Arabic (MSA), causing LLMs to overproduce MSA and struggle with dialectally accurate generation. From an interpretability perspective, this raises a fundamental question: where and how are dialectal features encoded within model internals, and can these representations be leveraged to improve dialect generation without fine-tuning? This study investigates two complementary inference-time approaches that serve simultaneously as interpretability probes and control mechanisms. First, we conduct a neuron-level analysis, identifying sparse neuron populations that encode dialect-specific features and showing that amplifying or suppressing these neurons can steer model outputs toward target dialects. Second, motivated by the entanglement of dialectal features at the single-neuron level, we apply a vector-steering approach that extracts dialect-specific activation directions and injects them during inference. Together, these methods illuminate the geometry of dialectal knowledge in Arabic LLMs and offer a principled, interpretability-grounded framework for dialect control without requiring dialect-specific fine-tuning.
Alex: If the neurons are individual switches, what's the residual stream in that picture?
Sam: Think of it as the main highway running through the model—the channel where all the processed information flows from one layer to the next as the model builds up its response. Instead of just flipping individual switches, the researchers took a different approach here. They showed the model examples of formal Arabic and examples of a target dialect, then measured the difference in how the model's internal state looked in each case. That difference—that gap—becomes a direction they can push the model in. Add that direction to the highway, and the whole flow of the model's thinking shifts toward the dialect they want. They call this a "steering vector."
Alex: So the individual switches are for fine-tuning specific details, but the steering vector is like changing the overall temperature of the language—everything shifts at once.
Sam: That's a good way to put it. And the vector-based approach turned out to be more reliable for producing consistent dialectal output. What it reveals is that dialectal identity inside these models isn't just a lookup table of regional words—it's a structured direction in the model's internal space that you can actually navigate.
Alex: But if you're pushing the model that hard in a new direction, doesn't it risk losing coherence? What if it starts speaking in dialect but stops making sense?
Sam: That's exactly the right concern, and the researchers took it seriously. If you push the steering vector too far, the model does start to lose its grip on meaning—the dialect becomes more pronounced, but the logic falls apart. So they carefully tested different strengths of the steering to find the point where the output sounds authentically dialectal but still makes sense. They used automated scoring to measure dialect accuracy, and also had a separate AI system evaluate whether the outputs remained coherent. The results confirmed that, at the right strength, the steering works without breaking the underlying intelligence of the model.
Alex: You mentioned something else—about the model actually understanding how these dialects relate to each other geographically?
Sam: Yes, and this is one of the more telling findings. When the researchers mapped out where different dialects sat in the model's internal space, they found that cities geographically close to each other—like Rabat and Tunis—also ended up close together in the model's internal map. The model wasn't explicitly taught that these cities are near each other. It learned that on its own, just from patterns in text. That suggests these models aren't simply memorizing strings of words—they're building something closer to an internal map of how human language varies across regions.
Alex: So the model has essentially taught itself a kind of hidden geography of Arabic dialects.
Sam: That's a fair way to describe it. And that's what makes this research meaningful beyond just the Arabic case. It shows that the internal structure of these models is far more organized than it might appear from the outside—and that organization can be used to correct biases and improve performance without the enormous cost of retraining. The dialect isn't a separate piece of software that needs to be installed. It's already a direction inside the model that we can learn to point toward.
Alex: That's a genuinely useful finding—not just for Arabic, but for any language where the formal written version dominates the training data while the spoken versions get left behind. Thanks for walking us through it, and thanks to everyone for listening to ResearchPod.