Climate-ModernBERT: Revisiting Corpus Composition for Domain-Adaptive Continued Pretraining

Climate-ModernBERT is a suite of encoder-only language models designed to improve performance on climate-related NLP tasks, such as risk assessment, disclosure analysis, and information retrieval. The researchers investigate how to best adapt the ModernBERT-Base architecture to the heterogeneous climate domain, which includes scientific literature, policy documents, and web discourse. They systematically compare two primary adaptation strategies: joint continued pretraining on mixed corpora and parameter-space merging of independently specialized checkpoints.

The authors curate a 6.42B-token climate corpus from three distinct sources: academic climate text (peer-reviewed papers and reports), climate-filtered web data, and synthetic climate documents generated by a large language model. They evaluate 21 model variants across nine climate NLP benchmarks. The study specifically compares joint training—where the model is trained on a mixture of these sources—against model merging, where separate checkpoints are trained on individual sources and then combined in parameter space using techniques like simple weight averaging (Soup) and TIES-Merging.

The study reveals that academic climate corpora provide the strongest and most transferable adaptation signal. Surprisingly, adding broader, less-curated sources like web or synthetic data to the academic corpus often leads to performance degradation on downstream tasks, suggesting that out-of-register data can interfere with domain-specific knowledge. Furthermore, the researchers find that parameter-space merging consistently outperforms joint multi-source training. The best-performing model, created via simple weight averaging of single-source checkpoints, achieves an average F1 score of 76.3, significantly outperforming both the vanilla ModernBERT baseline and joint-training variants. This suggests that merging preserves complementary information from heterogeneous sources more effectively than joint optimization.