José Cañete, Gabriel Chaperon, Rodrigo Fuentes, Jou-Hui Ho, Hojin Kang, Jorge Pérez
8 min
Despite Spanish being one of the world's most widely spoken languages, researchers have historically lacked dedicated, high-quality language models compared to English or Chinese. While multilingual BERT (mBERT) models exist, they are often outperformed by models trained exclusively on a single language. This study addresses this scarcity by developing and releasing the first BERT-based model pre-trained specifically on a large Spanish corpus.
The researchers pre-trained a Spanish BERT model with approximately 110 million parameters, adhering to the standard BERT-Base architecture. They trained two versions, one cased and one uncased, on a corpus of 3 billion words sourced from diverse materials including Wikipedia, government journals, and subtitles. To optimize performance, the training process integrated advanced techniques such as Dynamic Masking and Whole-Word Masking, ensuring that the model could handle the specific morphological structure of the Spanish language more effectively.
To provide a standardized evaluation framework, the authors compiled a collection of Spanish-specific tasks termed GLUES (GLUE for Spanish). This benchmark includes various language tasks such as Natural Language Inference, Paraphrasing, Named Entity Recognition, Part-of-Speech Tagging, Document Classification, Dependency Parsing, and Question Answering. By providing this resource, the study aims to establish a clear benchmark for future Spanish NLP development.
The study demonstrates that by fine-tuning their Spanish-only model on GLUES tasks, they achieve superior performance compared to existing multilingual BERT models in most categories, including new state-of-the-art results for POS tagging and Document Classification. The authors have publicly released the model, training data, and the benchmark repository, encouraging wider adoption and research in Spanish-speaking communities.
The Spanish language is one of the top 5 spoken languages in the world. Nevertheless, finding resources to train or evaluate Spanish language models is not an easy task. In this paper we help bridge this gap by presenting a BERT-based language model pre-trained exclusively on Spanish data. As a second contribution, we also compiled several tasks specifically for the Spanish language in a single repository much in the spirit of the GLUE benchmark. By fine-tuning our pre-trained Spanish model, we obtain better results compared to other BERT-based models pre-trained on multilingual corpora for most of the tasks, even achieving a new state-of-the-art on some of them. We have publicly released our model, the pre-training data, and the compilation of the Spanish benchmarks.
Sam: Two. The first is dynamic masking. Instead of hiding the same words in a sentence every time, each sentence got ten different masking patterns. The second is whole-word masking. The model reads text as subwords, pieces of words. With whole-word masking, if one piece of a word is hidden, all the pieces of that word are hidden together. So the model can't cheat by seeing the start of a word and guessing its ending.
Alex: So it's a familiar architecture with a more careful pre-training recipe, applied to one language. The novelty is the resource, not the method.
Sam: That's a fair reading. They also trained in two phases: short sequences with big batches first, then long sequences with smaller batches. That's an efficiency choice from earlier work.
Alex: Now the benchmark. What's in it?
Sam: They call it GLUES, a Spanish take on the English GLUE benchmark. It pulls together existing datasets. There's natural language inference: given two sentences, does the first imply, contradict, or neither? There's paraphrase detection, named entity recognition, part-of-speech tagging, and news document classification. There's dependency parsing, which builds a grammar tree for each sentence. And there's question answering, where you find the span of text that answers a question.
Alex: A lot of those sound translated from English. Does that matter?
Sam: It does, and they say so. For inference, for example, the training set is a machine translation of an English dataset. That point comes back in the question answering results.
Alex: Then let's get to the results. What's the headline comparison?
Sam: They compare against the best multilingual BERT score in the literature. The setting is fixed: training data in Spanish only. On inference, the multilingual model reached about 78.5 percent accuracy. Their cased Spanish model reached about 82. That's one of the largest gaps, and the authors note it's also the task with the biggest Spanish training set.
Alex: And the rest of the classification tasks?
Sam: Smaller wins mostly. On part-of-speech tagging and document classification, they report new state-of-the-art results. They call these two of the most standard Spanish datasets. On paraphrase and named entities, they edge ahead, though by different versions. The lowercased model actually did noticeably worse on named entities. That makes sense, since capital letters are a strong clue to names.
Alex: You mentioned question answering. How did that go?
Sam: Mixed. On one dataset called MLQA, training and testing on it, multilingual BERT scored about 54 on the overlap measure. The Spanish model scored about 68. But on the other two setups, it roughly tied, and on exact match it was sometimes worse.
Alex: A fourteen-point jump on one dataset and a tie elsewhere makes me wonder about the data, not the model.
Sam: The authors raise that too. They suggest MLQA's machine-translated training data may be low quality. They checked and found that nearly half its examples have a mismatch between the answer and where it supposedly starts in the text. So that gap may say as much about noisy data as about either model.
Alex: How much would you trust these numbers overall?
Sam: Moderately. The direction is consistent across most tasks. But some margins are small, under a point. The baselines are numbers reported by other papers, not rerun side by side. The tuning was a small standard grid of settings, and they leave a wider search to future work. They don't report variation across runs. And dependency parsing is described in the benchmark, but no results for it appear.
Alex: What's the biggest catch for someone deciding between this and a multilingual model?
Sam: That the comparison is deliberately narrow. Their model is Spanish only. It can't learn from the English training sets behind these translated benchmarks, or from data in other languages. Multilingual models can. The authors point to XLM-RoBERTa, a much bigger multilingual model with about 560 million parameters. Fine-tuned with data beyond Spanish, it reported 85.6 percent on inference, above their 82. It also beat them on named entities.
Alex: So the Spanish model wins when you only have Spanish labelled data. If you can borrow from English, a large multilingual model may still win.
Sam: That's what the paper's own numbers show. They give a similar case for paraphrase: multilingual BERT fine-tuned with the English training set added scored higher than their model.
Alex: So who should read the full paper, and where should they start?
Sam: Anyone building Spanish language tools, or training a single-language model for another language. Start with the GLUES section, because it tells you exactly which data splits to use. Then read the results discussion, especially the paragraph on what multilingual models gain from other languages. The training recipe section is short and worth a skim if you're pre-training your own.
Alex: And for everyone else, the one line to keep?
Sam: A model built for one language wins on that language's data alone. Give a multilingual model more languages to learn from, and it can win back the lead.
Alex: A useful reminder that "better" depends on what data you're allowed to use.