LangMAP: A Language-Adaptive Approach to Tokenization

Modern large language models (LLMs) often rely on tokenizers optimized for English, which leads to suboptimal segmentation for non-English languages and programming code. This over-segmentation increases computational costs and can degrade downstream model performance. The authors investigate whether it is possible to improve tokenization quality for specific languages without the prohibitive costs of retraining models from scratch or adapting their underlying vocabularies.

The authors introduce Language-adaptive Maximum a Posteriori (LangMAP) Tokenization. LangMAP extends the standard UnigramLM algorithm by learning separate probability distributions for different languages over a single, fixed vocabulary. During training, the algorithm uses language labels to estimate these per-language distributions. At inference time, LangMAP identifies the input language implicitly and selects the segmentation that maximizes the likelihood under the corresponding language-specific distribution. Because the vocabulary remains fixed, the method is compatible with existing pretrained models and avoids the need for new embeddings or auxiliary networks.

LangMAP demonstrates significant improvements in intrinsic tokenization metrics across 9 natural languages and 9 programming languages. Specifically, it enhances morphological boundary alignment for natural languages and improves alignment with abstract syntax tree (AST) leaf boundaries for code. In fine-tuning experiments, LangMAP shows clear benefits for grammatical acceptability (MultiBLiMP), particularly in agglutinative languages. However, these gains do not consistently translate to improved performance on knowledge-related benchmarks like Global-PIQA or Belebele, suggesting that while the segmentation is more linguistically accurate, it does not always improve higher-level reasoning capabilities.