Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, Ziqing Yang
8 min
Recent advances in natural language processing (NLP) have been driven by pre-trained language models like BERT. However, these models were primarily developed for English. This paper aims to adapt and evaluate pre-trained language models for the Chinese language. The authors construct a series of Chinese pre-trained models—including BERT-wwm, RoBERTa-wwm, and ELECTRA—and propose a new model called MacBERT to address specific limitations in the original pre-training process.
The authors identify a discrepancy between the pre-training and fine-tuning phases in standard BERT: the [MASK] token used during pre-training is never encountered during actual fine-tuning. To mitigate this, they propose a novel objective called MLM as correction (Mac). Instead of replacing tokens with an artificial [MASK] token, the model is tasked with predicting the original word from its surrounding context, where the masked tokens are replaced by their synonyms. This forces the model to learn a more natural 'correction' behavior. Additionally, the authors adopt Sentence Order Prediction (SOP) to replace the original Next Sentence Prediction (NSP) task, as SOP has been shown to be more effective for learning sentence relationships.
The researchers conducted extensive experiments across ten Chinese NLP tasks, ranging from document-level machine reading comprehension to sentence-level classification. MacBERT demonstrated consistent state-of-the-art performance across these tasks, particularly in reading comprehension, where it outperformed standard BERT, RoBERTa, and ELECTRA variants. Ablation studies confirmed that N-gram masking and the Mac task were the primary contributors to these gains, while the SOP task provided additional benefits for understanding longer-range dependencies.
This work provides the research community with robust, open-source Chinese pre-trained models, significantly lowering the barrier for entry in Chinese NLP. By demonstrating that the masking strategy itself—and specifically moving away from the artificial [MASK] token—can substantially improve performance, the paper provides a blueprint for future language-specific model development.
Bidirectional Encoder Representations from Transformers (BERT) has shown marvelous improvements across various NLP tasks, and its consecutive variants have been proposed to further improve the performance of the pre-trained language models. In this paper, we aim to first introduce the whole word masking (wwm) strategy for Chinese BERT, along with a series of Chinese pre-trained language models. Then we also propose a simple but effective model called MacBERT, which improves upon RoBERTa in several ways. Especially, we propose a new masking strategy called MLM as correction (Mac). To demonstrate the effectiveness of these models, we create a series of Chinese pre-trained language models as our baselines, including BERT, RoBERTa, ELECTRA, RBT, etc. We carried out extensive experiments on ten Chinese NLP tasks to evaluate the created Chinese pre-trained language models as well as the proposed MacBERT. Experimental results show that MacBERT could achieve state-of-the-art performances on many NLP tasks, and we also ablate details with several findings that may help future research. We open-source our pre-trained language models for further facilitating our research community.
Alex: Can you make that substitution tangible with the paper's example?
Sam: The English illustration starts with a sentence about using a “language model” to predict the next word. In the correction version, “language model” becomes “text system.” The model must recover the original wording from a sentence that still contains ordinary words.
Alex: Those replacements might preserve much of the meaning. Is the target just a plausible sentence, or the exact original text?
Sam: The original text. Similarity determines the corruption, not the answer. The authors obtain similar words using a toolkit based on learned word similarities; when a suitable replacement isn't available, they can fall back to a random word.
Alex: Does every selected word get this treatment, or do they mix different kinds of examples?
Sam: They select fifteen percent of input words for the exercise. Of the selected tokens, roughly four in five receive similar-word replacements. The remainder are either replaced randomly or left unchanged, so the model also encounters positions that need no correction.
Alex: And is correction the entire training recipe? Otherwise, an improvement could come from another change.
Sam: There's also sentence order prediction: deciding whether consecutive passages have been swapped. MacBERT builds on RoBERTa, a BERT variant with a revised training recipe. So the headline comparison tests a package; the authors also remove components separately to examine their contributions.
Alex: Let's hear that headline evidence first. How broadly did they test the package?
Sam: The main evaluation covers ten Chinese datasets, spanning reading comprehension, single-text classification, and sentence-pair classification. They repeated each experiment ten times and reported both peak and average performance. That helps distinguish a model's best run from its more typical result.
Alex: Which comparison gives the clearest sense of the gain, without comparing against a much weaker starting point?
Sam: On the Traditional Chinese reading dataset DRCD, large MacBERT averaged about 95.3 on the test-set F1 score. The comparable large RoBERTa model averaged 94.1. F1 balances precision and recall; here, it measures overlap between predicted answers and reference answers.
Alex: That sounds useful, but it isn't evidence that every task improves equally. Where does the advantage become less clear?
Sam: Classification gains were generally smaller, especially on datasets where accuracy was already high. MacBERT wasn't best on every measure. Even on the difficult reading-comprehension challenge set, its advantage over large RoBERTa was slight.
Alex: What about training data? A better score is harder to interpret if the new model simply saw much more text.
Sam: The key RoBERTa and MacBERT comparisons use the same extended data collection. It includes encyclopedia, news, and question-answering material. But comparisons against original Chinese BERT combine method changes with a substantial data increase, so those aren't clean estimates of the correction strategy alone.
Alex: Then the component removals matter. Do they point toward correction, or toward the other changes?
Sam: Removing any component lowered the reported overall average. Selecting neighboring stretches and replacing words with similar words were the most effective modifications. The former seemed more helpful for classification; the latter seemed more helpful for reading comprehension.
Alex: That supports the recipe. Does it establish that removing the artificial mask symbol is why it works?
Sam: Not by itself. A more targeted experiment compared similar-word replacement, random replacement, and mask-based alternatives on reading comprehension. Similar words performed best, while random replacement also improved over the mask-based strategies.
Alex: So the results favor ordinary replacement words, but there could be more than one explanation. What alternatives do the authors acknowledge?
Sam: They suggest random replacements might improve denoising ability: learning to recover text from corruption. Similar words may also preserve more natural phrasing. My reading is that these comparisons support the mismatch explanation, but don't isolate it from those other benefits.
Alex: Is there another design detail researchers should keep in mind before treating the rankings as definitive?
Sam: Learning rates were generally chosen using original Chinese BERT, rather than optimized separately for every model. The authors acknowledge that individual tuning might yield further gains. And the main benchmarks are Chinese, so this study doesn't establish equivalent benefits across languages.
Alex: Who should read the full document, and where should they begin?
Sam: Researchers building Chinese language systems should start with the MacBERT method section, then “Investigation on MLM Task.” That replacement-strategy comparison is the clearest test of the central idea. For choosing a released model, inspect the average results on tasks resembling your own, rather than just the best score.
Alex: And what should everyone else remember once the model names have faded?
Sam: How you damage the training text helps determine what the model learns to repair. Here, replacing blanks with plausible wrong words improved many results—but the gains depended on the task.
Alex: Thanks for listening.