Hunter Heidenreich
5 min
Chemical language models typically use tokenizers inherited from natural language, most commonly Byte-Pair Encoding (BPE). While BPE and Unigram-LM are known to produce different vocabularies in natural language, it was unclear if this contrast persists in chemistry. Chemical SMILES strings have a tiny, fixed alphabet and are subject to strict valence constraints, which could theoretically force both algorithms to converge on the same high-frequency substructures. This study performs a controlled comparison of BPE and Unigram-LM across three corpus typologies (diverse, drug-like, and natural-products) and two pre-tokenization boundary policies, using a fixed 165-token chemistry base to isolate the subword algorithm as the sole design variable.
The two algorithms do not converge. In all 22 matched conditions, BPE and Unigram-LM build near-disjoint vocabularies, with cross-algorithm Jaccard overlap never exceeding 0.161. This divergence deepens when weighted by the frequency of pieces a model actually encounters. Furthermore, Unigram-LM consistently produces finer segmentations, resulting in 29-41% more tokens per molecule than BPE. Despite these differences, the two algorithms are largely compatible: BPE’s segmentation acts as a strict coarsening of Unigram-LM’s in 80-99% of molecules, meaning BPE essentially merges the same substructures that Unigram-LM keeps atomic. This separation is stable across corpus, boundary policy, and vocabulary size, persisting even at eight times the headline vocabulary scale.
These results demonstrate that the subword algorithm is a consequential design choice rather than a free default. BPE favors shorter sequences, which can reduce computational costs and extend effective context length, while Unigram-LM retains finer-grained pieces and supports principled subword regularization. Because the vocabularies transfer across domains with near-native efficiency, researchers should treat the choice of tokenizer as a deliberate modeling decision that shapes the granularity at which a model perceives chemical structure.
Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE's principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that contrast survives in chemistry was open. We report a controlled comparison of BPE and Unigram-LM over a fixed 165-token chemistry base, at the small vocabulary sizes where token embeddings are learnable, across three corpus typologies (diverse, drug-like, natural-products) and both pre-tokenization boundary policies. The two do not converge. In all 22 matched conditions they build near-disjoint subword vocabularies: cross-algorithm Jaccard overlap on the learned pieces never exceeds 0.161, and at most 0.05 once weighted toward the high-frequency pieces a model updates most. Unigram-LM also segments held-out molecules into 29-41% more tokens; the arms largely agree on where to cut but not how deeply, so BPE's segmentation is a strict coarsening of Unigram-LM's on 80-99% of molecules. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale. The subword algorithm is therefore a modeling decision, not a free default. The study trains no language models.
Alex: So the "how" of tokenization isn't just a technical detail—it's a foundational choice that shapes everything downstream.
Sam: Precisely. The authors conclude that the choice of subword algorithm is a genuine modeling decision, not a free default you can ignore. And this divergence isn't a quirk of one particular dataset or one particular vocabulary size—it persists across every condition they tested. It's a structural difference baked into how these algorithms approach chemistry.
Alex: That raises a question for me. If you keep adding more and more pieces to the vocabulary, doesn't the model eventually start learning pieces it almost never sees? Like studying vocabulary words you'll never actually use?
Sam: That's a real concern, and the researchers looked at it directly. They measured something they call "clearance"—essentially, how often a learned piece actually shows up in the training data. What they found is that as the vocabulary grows larger, many pieces become what they call "undertrained." They appear so rarely that the model never gets enough examples to build a reliable understanding of what that piece means chemically.
Alex: So there's a natural ceiling on how useful a bigger vocabulary can be.
Sam: Right. And this connects to something specific about chemistry versus human language. When you write English, there are countless ways to combine words and phrases, so a large vocabulary of common chunks makes sense. But chemical formulas follow strict rules—there are only so many valid ways to arrange atoms. That means the space of high-frequency patterns is much more constrained. The "best" way to slice up a molecule isn't obvious, and the paper argues it's a question that deserves much more careful thought than it's currently getting.
Alex: So the broader implication is that researchers building chemical AI models may have been treating tokenization as a solved problem when it's actually still an open one.
Sam: That's the paper's central argument. By carefully isolating the tokenization algorithm from everything else—the dataset, the vocabulary size, the model architecture—the researchers show that this one choice alone produces fundamentally different internal representations. It's a call for more scrutiny at the very first step of model design, before any training even begins.
Alex: It's one of those findings that seems obvious in retrospect, but only once someone has actually done the work to demonstrate it clearly.
Sam: Exactly. The tools we borrow from one field don't always transfer cleanly to another. Chemistry has its own logic, its own constraints, and its own structure. The paper suggests that the field of chemical AI would benefit from building tokenization methods that are designed with that chemistry in mind, rather than inherited by default.
Alex: Thanks for walking us through that, Sam. And thanks to everyone listening to ResearchPod.