Predicting Multilingual Classification and Translation Performance of LLMs with Cross-Lingual Alignment – Is English Enough?

This paper investigates how cross-lingual alignment (CLA) scores—metrics that quantify how similarly different languages are represented within a model—predict the performance of multilingual Large Language Models (LLMs) on downstream tasks. While prior research has established that high alignment with English correlates with better performance on classification and reading comprehension, this study extends that inquiry to machine translation. The authors analyze 27 different CLA score variants across three tasks (SIB-200, Belebele, and Flores-200) to determine whether source-to-target alignment or source-to-English alignment is more predictive of translation quality.

To conduct this analysis, the researchers evaluated three base and three instruction-tuned LLMs (Qwen3, Gemma-3, and Ministral-3). They introduced a novel 'few-shot' method for extracting sentence embeddings that avoids the bias of English-language prompts. Because standard translation metrics like chrF are sensitive to the target language's writing system, the authors developed a new translation evaluation metric based on Pointwise Mutual Information (PMI). This metric is designed to be less dependent on the target language, allowing for more robust correlation analysis across diverse language pairs.

The results demonstrate that CLA scores with English are consistently strong predictors of performance across all tested tasks. In the context of machine translation, the authors found that alignment with English is often more predictive than direct source-to-target alignment. This finding supports the hypothesis that LLMs implicitly use English as an internal pivot language, even when performing direct translation between two non-English languages. The study also provides a comprehensive review of embedding extraction techniques, suggesting that the choice of sentence representation (e.g., position-weighted mean) and alignment metric (e.g., ANC) significantly impacts the predictive power of these scores.