IHUBERT: Vector-Based Semantic Deduplication and Domain-Balanced Pretraining for Persian Resources

Existing Persian pretrained language models (PLMs) have often been limited by the quality and diversity of their pretraining data, as well as a narrow focus on simple classification and named entity recognition (NER) tasks. The authors introduce IHUBERT, a RoBERTa-base model trained from scratch on a 45GB curated subset of the Sepahr-Danesh collection. The study addresses data quality through a multi-stage pipeline that includes normalization, anonymization, and a novel vector-database-based semantic deduplication method to balance domain representation. Additionally, the authors trained a 139k-vocabulary Byte-Pair Encoding (BPE) tokenizer specifically to handle Persian's complex morphology and orthographic variations.

IHUBERT demonstrates significant improvements in reasoning and comprehension tasks. It achieved top rankings on the PQuAD and ParsiNLU-RC extractive question answering datasets, as well as the FarsTail natural language inference benchmark. In standard classification and NER tasks, the model remains highly competitive with existing state-of-the-art Persian models. The authors also conducted a controlled ablation study comparing BPE to WordPiece, finding that their chosen BPE configuration resulted in lower subword fragmentation, which supports the model's ability to process diverse formal and informal Persian text.

This work highlights the importance of data curation over raw scale. By implementing semantic deduplication and a larger, morphology-aware vocabulary, the authors demonstrate that a medium-sized model (125M parameters) can outperform previous baselines on complex NLU tasks. This provides a robust foundation for future Persian NLP research, particularly for applications requiring deep semantic understanding rather than just surface-level classification.