MoganBert-TR: A Turkish Encoder Foundation Model Trained from Scratch with a CLM→MLM Curriculum

Modern encoder-only models for Turkish have largely adopted updated architectures (like ModernBERT) while retaining the traditional masked language modeling (MLM) pretraining objective. This paper investigates whether the pretraining objective itself can be optimized for Turkish, a morphologically rich and agglutinative language, and how to effectively manage the annealing phase for long-context extension.

The authors introduce MoganBert-TR, a 149M-parameter encoder trained from scratch on a high-quality, language-specifically filtered Turkish corpus. The training utilizes a two-stage curriculum: Causal Language Modeling (CLM) followed by Masked Language Modeling (MLM). This design is compared against a pure MLM baseline under an equal compute budget. Furthermore, the authors implement a 'branched annealing' strategy, where the final learning-rate decay phase is split into two branches—one at 1024 context and one at 8192 context—to test the impact of context length on downstream performance. Finally, the authors derive an embedding model, MoganBert-Embed, through teacher distillation and multi-signal contrastive fine-tuning.

The CLM-to-MLM curriculum significantly improves embedding geometry compared to pure MLM. In pure MLM models, a single 'rogue dimension' often absorbs a large portion of the variance, compressing vectors into a narrow cone and degrading retrieval performance. The curriculum reduces this variance, leading to a 2.7–3.7x improvement in retrieval metrics. The branched annealing approach proved highly effective, with the 1024-context branch outperforming a model-soup alternative by 0.75 points on TrGLUE at a fraction of the cost. MoganBert-TR achieves the best performance among compared Turkish ModernBERT models on TrGLUE (78.41) and shows a notable lead in code retrieval (+3.62 points over TabiBERT), attributed to the indentation-preserving tokenizer and a high code share in the pretraining mixture. The derived embedding model, MoganBert-Embed, reaches 99.5% of its 7.57B-parameter teacher's performance while being 51x smaller.