Hunter Heidenreich
5 min
Abstract
Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE's principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that contrast survives in chemistry was open. We report a controlled comparison of BPE and Unigram-LM over a fixed 165-token chemistry base, at the small vocabulary sizes where token embeddings are learnable, across three corpus typologies (diverse, drug-like, natural-products) and both pre-tokenization boundary policies. The two do not converge. In all 22 matched conditions they build near-disjoint subword vocabularies: cross-algorithm Jaccard overlap on the learned pieces never exceeds 0.161, and at most 0.05 once weighted toward the high-frequency pieces a model updates most. Unigram-LM also segments held-out molecules into 29-41% more tokens; the arms largely agree on where to cut but not how deeply, so BPE's segmentation is a strict coarsening of Unigram-LM's on 80-99% of molecules. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale. The subword algorithm is therefore a modeling decision, not a free default. The study trains no language models.
Alex: So the "how" of tokenization isn't just a technical detail—it's a foundational choice that shapes everything downstream.
Sam: Precisely. The authors conclude that the choice of subword algorithm is a genuine modeling decision, not a free default you can ignore. And this divergence isn't a quirk of one particular dataset or one particular vocabulary size—it persists across every condition they tested. It's a structural difference baked into how these algorithms approach chemistry.
Alex: That raises a question for me. If you keep adding more and more pieces to the vocabulary, doesn't the model eventually start learning pieces it almost never sees? Like studying vocabulary words you'll never actually use?
Sam: That's a real concern, and the researchers looked at it directly. They measured something they call "clearance"—essentially, how often a learned piece actually shows up in the training data. What they found is that as the vocabulary grows larger, many pieces become what they call "undertrained." They appear so rarely that the model never gets enough examples to build a reliable understanding of what that piece means chemically.
Alex: So there's a natural ceiling on how useful a bigger vocabulary can be.
Sam: Right. And this connects to something specific about chemistry versus human language. When you write English, there are countless ways to combine words and phrases, so a large vocabulary of common chunks makes sense. But chemical formulas follow strict rules—there are only so many valid ways to arrange atoms. That means the space of high-frequency patterns is much more constrained. The "best" way to slice up a molecule isn't obvious, and the paper argues it's a question that deserves much more careful thought than it's currently getting.
Alex: So the broader implication is that researchers building chemical AI models may have been treating tokenization as a solved problem when it's actually still an open one.
Sam: That's the paper's central argument. By carefully isolating the tokenization algorithm from everything else—the dataset, the vocabulary size, the model architecture—the researchers show that this one choice alone produces fundamentally different internal representations. It's a call for more scrutiny at the very first step of model design, before any training even begins.
Alex: It's one of those findings that seems obvious in retrospect, but only once someone has actually done the work to demonstrate it clearly.
Sam: Exactly. The tools we borrow from one field don't always transfer cleanly to another. Chemistry has its own logic, its own constraints, and its own structure. The paper suggests that the field of chemical AI would benefit from building tokenization methods that are designed with that chemistry in mind, rather than inherited by default.
Alex: Thanks for walking us through that, Sam. And thanks to everyone listening to ResearchPod.