Uri Katz, Omer Goldman, Tomasz Limisiewicz, Reut Tsarfaty, Noah A. Smith
5 min
Large language models (LLMs) often perform poorly on low-resource languages due to a reliance on noisy, web-scale data that frequently contains machine-translated spam or misclassified text. Yiddish, despite its rich literary tradition, suffers from a limited digital footprint and a lack of reliable evaluation resources. This paper addresses these gaps by building a dedicated language model and a rigorous evaluation framework.
The authors introduce two primary resources to support Yiddish NLP:
Using these resources, the authors performed continued pretraining of the Llama 3.1 8B model. To maintain general capabilities, they included a mixture of English data during training, while specifically excluding benchmark data to prevent leakage.
MameLoshnLM consistently outperforms open-weight baselines of similar scale across the Kashes benchmark. Beyond quantitative metrics, qualitative analysis suggests that the model produces more natural Yiddish, avoiding the "translationese" and overly Germanized patterns common in general-purpose multilingual models. The study also demonstrates that while English data is helpful for retaining base model capabilities, adding data from genealogically related languages (like German or Hebrew) provides only marginal benefits.
This work provides a practical template for developing language models for historically rich but digitally underrepresented languages. It highlights that for such languages, the primary obstacle is not just data volume, but the quality and authenticity of the training data. By creating curated corpora and targeted benchmarks, researchers can significantly improve model performance and linguistic accuracy.
We present MameLoshnLM, the first open-source 8B-parameter language model built specifically for Yiddish. Despite Yiddish's rich textual tradition, its limited digital presence and the scarcity of reliable evaluation resources have constrained progress in Yiddish language modeling. Existing multilingual corpora and benchmarks are often poor proxies for the language, containing substantial amounts of noisy, machine-translated, and misclassified text. We address these gaps by introducing Oytser, a high-quality Yiddish pretraining corpus that combines contemporary web-native sources with literary materials, and Kashes, a multi-task benchmark spanning translation, linguistic analysis, information extraction, and language understanding. Using these resources, we continue pretraining Llama 3.1 8B to obtain MameLoshnLM. Across the tasks in the benchmark, MameLoshnLM outperforms open baselines of similar scale. Our analyses show that these gains are not only quantitative: relative to general-purpose multilingual models, MameLoshnLM better captures language-defining lexical and morphological patterns, pointing to a broader failure mode of noisy web-scale multilingual data for low-resource languages. Our results provide both a foundation for Yiddish NLP and a practical template for language model development in historically rich but digitally underrepresented languages.
Sam: And then they didn't train a model from scratch on this—they took an existing model and continued training it on this new, cleaner data?
Alex: Correct. The technique is called "continued pretraining." Think of it like this: imagine a student who has already learned to read and write well in several languages. Rather than starting their education over, you enroll them in a rigorous, high-quality literature course focused specifically on Yiddish. They're building on what they already know, but now they're absorbing authentic material instead of rough approximations.
Sam: And did it work? Did the model actually start picking up those loshn-koydesh patterns that the noisy data had washed out?
Alex: The paper reports that it did. The model successfully recovered those linguistic patterns—the Hebrew and Aramaic-origin vocabulary used in context the way a native speaker would use it. That's a meaningful result, because it suggests the underlying capacity was always there. The model just needed authentic material to learn from.
Sam: That's a striking way to put it. The model wasn't broken—it was just being fed the wrong diet.
Alex: That's a fair summary of the paper's argument. For languages that are historically rich but digitally underrepresented, the core obstacle isn't a lack of computing power or model sophistication. It's the mismatch between how the language is actually used by real speakers and what ends up indexed on the public web.
Sam: Which raises a question about how you even verify improvement. How do you know the model is genuinely better at Yiddish, rather than just appearing to be?
Alex: That's where the benchmark comes in. The researchers also developed something called "Kashes"—a structured set of evaluation tasks designed specifically for Yiddish. Without a benchmark like this, you're essentially guessing. Kashes gives researchers a consistent, reproducible way to measure whether a model is actually performing better on real tasks, not just on paper.
Sam: So the contribution here is really threefold: a cleaner dataset, a retrained model, and a proper evaluation framework.
Alex: Precisely. And the paper is candid about the trade-offs. A model trained this way excels at Yiddish, but it doesn't have the broad general reasoning power of a model trained on trillions of tokens across dozens of languages. It's a specialized instrument, not a general-purpose one.
Sam: That seems like an honest framing. It's not trying to replace everything—it's trying to do one thing well.
Alex: And the broader implication the paper points to is that this approach could serve as a template. There are many languages in a similar position to Yiddish—historically significant, culturally rich, but poorly represented in the data that powers modern AI. The argument is that careful curation of authentic sources, combined with continued pretraining, is a scalable path toward making those languages visible again. Thanks for listening to ResearchPod.