Daryna Dementieva, Nikolay Babakov, Kathy Hämmerl, Ilseyar Alimova, Jindřich Libovický, Shu Okabe, Miras Baisbay, Lukas Edman, Abrorkhon Inomkhujaev, Antonia Karamolegkou, Mateusz Lango, Volkan Özer, Nikola Selic, Subhankar Swain, Tsedeniya Kinfe Temesgen, Galit Bary Weisberg, Alexander Fraser
5 min
Abstract
Mathematical reasoning has become a central task for evaluating and tuning reasoning Large Language Models (LLMs), yet existing benchmarks remain heavily biased toward high-resource languages, with English and Chinese dominating both pre-training corpora and evaluation suites. The recently released PolyMath (Wang et al., 2025) dataset represents a significant step forward, yet its coverage is still limited to 18 only high-resource languages. To address this gap, we introduce PluraMath, an extension of PolyMath to 18 additional {underrepresented languages spanning 6 language families -- ranging from mid-resource to extreme low-resource settings. We constructed the dataset through a human-curated pipeline, where native speakers thoroughly validated pre-computed translations. Using PluraMath, we then benchmark 27 reasoning LLMs across four model scales -- small, mid-size, large, and closed-source ensembles -- probing the multilingual mathematical reasoning capabilities of state-of-the-art models under diverse linguistic conditions. Our fine-grained analysis confirms a persistent gap in mathematical reasoning performance between high-resource and underrepresented languages, with stronger results largely associated with better instruction-following ability. We fully open-source our dataset, data acquisition pipeline, and evaluation framework, with the goal of lowering the barrier to multilingual benchmark development for underrepresented communities.
Sam: That's the interesting part to me. If the translations were verified and accurate, why were the models still struggling?
Alex: That's exactly what the study digs into. You might assume the issue is translation quality—that the AI is tripping over unfamiliar words or awkward phrasing. But the evidence points somewhere else. The researchers found that how well a model performs in a given language is only weakly connected to how well it can translate text in that language. What it's much more strongly connected to is something called general instruction-following ability—how well the model understands and responds to directions overall.
Sam: So it's less like the model is confused by the foreign words, and more like... it loses its footing when the instructions aren't in the language it was primarily trained on?
Alex: That's a good way to put it. Think of it like this: imagine you're a strong chess player, but you learned all the rules and strategies from a coach who only ever spoke to you in English. If someone suddenly gives you instructions mid-game in a language you barely know, you might still remember how the pieces move—but you'd struggle to follow the game plan. The underlying skill is there, but accessing it becomes harder when the instructions don't match your primary training.
Sam: That's a meaningful distinction. It suggests that just translating more training data isn't necessarily the whole solution—the models need to learn to follow instructions fluently across languages, not just recognize words in them.
Alex: That's what the study implies, yes. And it points to a real challenge for making AI tools genuinely useful on a global scale. The researchers also note that for languages with extremely limited written resources online, even this approach has limits—fully manual translation would be required, which is significantly more costly and time-consuming.
Sam: And I imagine there's also a question of whether the math problems themselves are equally relevant across different educational systems and cultures.
Alex: The study raises exactly that concern. Mathematical curricula aren't identical around the world, and a problem that feels standard in one context might be unfamiliar or framed differently in another. The researchers acknowledge this as an open question.
Sam: So PLURAMATH is less a final answer and more a careful first step—a way of making the problem visible so it can actually be addressed.
Alex: That's a fair characterisation. By building a verified, multilingual benchmark, the study gives researchers a clearer picture of where these gaps exist and what's actually driving them. That kind of diagnostic work is necessary before meaningful improvements can be made. Thanks for listening to ResearchPod.