CULTIVAR: A Contrastive and Locale-Oriented Translation Benchmark for Investigating Contamination and Localisation Robustness

Most multilingual translation benchmarks are constructed by translating English source text into other languages, creating a "language-pair" evaluation paradigm. This approach is increasingly prone to data contamination and fails to account for cultural and regional variations within a single language. The authors ask: how do translation models perform when evaluated on culturally localized content, and can we quantify the robustness of these models against benchmark contamination?

The authors introduce CULTIVAR, a new benchmark consisting of 200 named-entity-rich sentences for each of 27 language-locale combinations. They use a three-stage pipeline: identifying entity-rich instances from the FLORES dataset, using an LLM to paraphrase these instances to reflect specific cultural contexts (e.g., changing a U.S. university to a German one), and conducting human post-editing to ensure quality. By comparing model performance on the original FLORES instances versus the localized CULTIVAR instances, the authors measure "localization robustness" through performance discrepancies (delta metrics).

Testing 32 open-weight models, the researchers found that machine translation-specialized models are often less robust to localization than general-purpose LLMs. They observed significant performance gaps between the original FLORES test sets and the localized CULTIVAR sets, suggesting that some models may have overfitted to the original FLORES data. Furthermore, models consistently demonstrated a performance bias, translating content grounded in the United States more accurately than content localized to other regions, regardless of the target language.

This work highlights a critical blind spot in current machine translation evaluation. By moving from static, language-oriented benchmarks to dynamic, locale-oriented ones, researchers can better assess how models handle real-world cultural nuances. The findings suggest that current "state-of-the-art" performance may be inflated by data contamination and a lack of cultural diversity in test sets, indicating a need for more robust, localized evaluation standards.