Mark Schutera
7 min
Small language model (SLM) research often relies on tiny, curated datasets like Karpathy's tiny_shakespeare to prototype architectures and fine-tuning pipelines. While German literary resources exist, they are often fragmented, requiring significant parser engineering—such as handling TEI/XML, normalizing speaker tags, and fixing encoding issues—before they can be used. This "single-file friction" creates a barrier for researchers and students who want to work with German literary registers without spending hours on data preprocessing.
TINY_SCHILLER addresses this gap by providing a 2.07 MB, single-file collection of eleven public-domain dramas by Friedrich Schiller. The corpus is processed through a deterministic pipeline that normalizes formatting and unifies speaker tags into a standard format compatible with popular SLM tools like nanoGPT. By removing the need for custom parsers, the author enables researchers to load the data with a single line of code, facilitating immediate experimentation.
The corpus includes multiple precomputed tokenization splits (character-level, GPT-2 BPE, and cl100k_base), allowing for direct comparison of tokenization efficiency on German text. Additionally, the release includes instruction-formatted Parquet files and 89 per-character persona splits, which support supervised fine-tuning (SFT) workflows. The author demonstrates the utility of these artifacts with a reference two-stage SFT pipeline on a consumer GPU, showing that stylistic and persona-specific adaptation can be achieved in a single, short session.
TINY_SCHILLER lowers the barrier to entry for non-English language modeling, providing a reproducible testbed for studying how models handle the specific orthographic and stylistic features of German drama. By standardizing the data ingestion process, it allows practitioners to focus on modeling and evaluation rather than data cleaning, effectively extending the "tiny corpus" paradigm to the German language.
tiny_schiller closes the small-language-model prototyping, fine-tuning, education, and research gap for German literary text, providing a single-file, drop-in counterpart to Karpathy's tiny_shakespeare. The available German literary corpora are larger and richer, but require parser engineering before a single line of training or fine-tuning code can run. tiny_schiller is a 2.07-megabyte single file of eleven public-domain Schiller dramas, sourced from DraCor's GerDraCor export (CC0) and processed by deterministic parser engineering. Character-level, GPT-2 byte-pair encoding, and cl100k_base tokenization splits, an instruction-formatted dialogue-completion split, and 89 per-character persona splits load from a single HuggingFace call. A small language model literally reaches German literary text in one line of code.
Sam: So it's about accessibility. If you need a supercomputer to train a model, only a handful of well-funded institutions can do it. But if you can do it on a laptop, anyone can learn.
Alex: Precisely. And that's the core philosophy here — being operational and practical rather than aiming for perfection.
Sam: So how did he actually clean this data? What does that process look like in practice?
Alex: He used a fixed set of rules to process the text — converting all the different ways of labeling speakers into one single, consistent format. Think of it like forcing everyone in a group chat to use the same username style. He also fixed common issues like line breaks and special characters. By doing this once and releasing the result, he saves every other researcher from having to solve those same problems themselves.
Sam: What about tokenization? I know models don't read words the way we do — they break text into smaller pieces called "tokens." How does that work here?
Alex: Right. Tokens are the building blocks of text — sometimes whole words, sometimes just parts of words, like syllables. The author provides three different ways to break the text into these blocks, including a character-level approach and two standard subword methods.
Sam: Why three? Does it actually matter which one you use?
Alex: It matters quite a bit. English-trained tokenizers often struggle with German because they aren't optimized for its specific grammar or vocabulary. German, for instance, builds long compound words that an English-trained system might chop up inefficiently — using more tokens than necessary, which makes the model slower and more expensive to run. By offering three different splits, the author lets researchers test how each approach affects their results on the same consistent dataset.
Sam: So it's not just a file of text — it's a testbed for understanding how language models handle German specifically.
Alex: That's a fair way to put it. It serves multiple purposes, from rapid prototyping to deeper technical analysis.
Sam: You mentioned the author also included pre-formatted files for something called "supervised fine-tuning." What's that?
Alex: Fine-tuning is when you take a model that already knows how to generate language and teach it to behave in a more specific way — like following instructions, or adopting the voice of a particular character. Instead of feeding the model raw text, you give it paired examples: a prompt and the expected response. The author created these pre-formatted files so you can immediately start training a model to speak like a character from a Schiller play, for instance, without having to build that data structure yourself.
Sam: And the author actually tested this? It wasn't just a theoretical claim?
Alex: He did. He showed that a model with around half a billion parameters — which is small by modern standards — could be fine-tuned to adopt a specific persona in just a few minutes of compute time. It's a practical demonstration that the "drop-in" promise holds up.
Sam: What are the limitations? Is it perfect?
Alex: No, and the author is clear about that. The cleaning process is based on rules, which means some edge cases might still be messy. If a stage direction is formatted unusually, it might slip through. It's a trade-off: you get a clean, usable file for research, but you sacrifice the absolute precision of a hand-edited literary archive.
Sam: That seems like a reasonable trade-off. Better to have something usable today than a perfect version that never gets finished.
Alex: That's exactly the author's position. And there's also a structured summary included — similar to a nutrition label — that tells researchers exactly what's in the file, where it came from, and how it's intended to be used. It makes the data self-documenting, which reduces friction further for anyone picking it up for the first time.
Sam: One thing I keep wondering — why Schiller specifically? Why not Goethe, or someone else?
Alex: The author chose Schiller because his works are stylistically consistent, use a rich variety of German vocabulary, and are all in the public domain. That combination makes it a reliable testbed. The author also suggests the same approach could be applied to other German playwrights in the future, potentially building out a whole library of these "TINY_AUTHOR" corpora for comparative studies across different styles.
Sam: So this is less about Schiller himself and more about establishing a template that others can follow.
Alex: Exactly. It's a foundational contribution — not about building something massive, but about removing the barriers that prevent others from doing their own work. Sometimes that's the most useful thing a piece of research can do.
Sam: It's a good reminder that infrastructure work — the quiet, unglamorous kind — is what makes the more visible research possible.
Alex: Well put. Thanks for listening to ResearchPod.