M-GATE: Multilingual Grammar, Accuracy in Translation, and Efficiency Benchmark for Large Language Models

Modern large language models (LLMs) are frequently deployed across dozens of languages, yet their performance is often evaluated using tasks that conflate fluency with true linguistic proficiency. A model might generate fluent text in a language like Finnish while simultaneously failing to identify a basic grammatical error that a native-speaking child would easily catch. This discrepancy arises because most existing benchmarks test whether a model can perform a task (like reasoning or knowledge retrieval) in a language, rather than whether it possesses an internal command of that language's grammatical rules and structural constraints.

The M-GATE (Multilingual Grammar, Accuracy in Translation, and Efficiency) benchmark is designed to isolate and measure this linguistic proficiency. It covers 30 typologically diverse languages, ranging from high-resource languages like English and Chinese to low-resource languages like Kinyarwanda and Dzongkha. The benchmark employs two primary tasks:

1. Grammatical Error Detection: Professional linguists crafted adversarial sentences that turn on language-specific phenomena absent from English. These items are designed to be difficult, forcing the model to rely on its internal understanding of the target language's grammar rather than surface-level statistical patterns.
2. Round-Trip Translation: Models translate English source sentences into the target language and then back into English. This approach allows for automated scoring by comparing the original and final English outputs, bypassing the need for human reference translations or judges that are more capable than the model being tested.

The study evaluated over 50 models and found that translation quality is strongly correlated with a language's share of pretraining data. However, grammatical proficiency does not scale as reliably. Many models that perform well on translation tasks perform near chance on the adversarial grammar items, demonstrating a systematic bias toward under-flagging errors—essentially accepting ungrammatical text as correct. Furthermore, while enabling reasoning capabilities generally improves translation performance, its effect on grammar detection is inconsistent and occasionally detrimental, suggesting that the optimal configuration for a model is highly task-dependent.

AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.