Every chemical language model reading SMILES begins with a tokenizer, yet the field has inherited byte-pair encoding (BPE) from natural language with little scrutiny. In natural language, BPE's principal alternative, Unigram-LM, is known to build structurally different vocabularies. Whether that contrast survives in chemistry was open. We report a controlled comparison of BPE and Unigram-LM over a fixed 165-token chemistry base, at the small vocabulary sizes where token embeddings are learnable, across three corpus typologies (diverse, drug-like, natural-products) and both pre-tokenization boundary policies. The two do not converge. In all 22 matched conditions they build near-disjoint subword vocabularies: cross-algorithm Jaccard overlap on the learned pieces never exceeds 0.161, and at most 0.05 once weighted toward the high-frequency pieces a model updates most. Unigram-LM also segments held-out molecules into 29-41% more tokens; the arms largely agree on where to cut but not how deeply, so BPE's segmentation is a strict coarsening of Unigram-LM's on 80-99% of molecules. The separation holds across corpus, boundary, and vocabulary size, persisting even at eight times that scale. The subword algorithm is therefore a modeling decision, not a free default. The study trains no language models.
Alex: Welcome to another episode of ResearchPod. Today, we're looking at a study that challenges how we build AI for chemistry. Sam, what's the core puzzle here?
Sam: We're discussing how computers "read" chemical formulas. Chemists have a way of writing molecules as text strings—a kind of shorthand where the letters and symbols spell out a molecule's structure. These are called SMILES strings. The puzzle the paper tackles is whether the method used to break these strings into smaller pieces is a neutral technical choice, or whether it's actually a significant design decision that changes how a model understands chemistry.
Alex: So this paper is basically asking: have we just been copying tools from language AI without checking if they actually fit chemistry?
Sam: That's exactly it. Most chemical AI models inherit a method called Byte-Pair Encoding—BPE—from the world of language models. The way BPE works is fairly intuitive: it scans through text and repeatedly merges the two characters or character groups that appear together most often. Do that enough times, and you end up with a vocabulary of common "chunks." The paper tests this against a different approach called Unigram-LM, which works more like a probability puzzle—it figures out the most statistically likely way to break a string apart, rather than just grabbing the most frequent pairs.
Alex: Let me make sure I follow. If I'm learning a language, I might group letters into common words—like recognising "ing" or "tion" as a unit. Is that what these algorithms are doing with chemical symbols?
Sam: That's a good way to think about it. Imagine building with Lego. BPE is like a factory that pre-assembles large, common kits—so you get whole sections already snapped together. Unigram-LM is more like keeping individual bricks and only joining them when the math says it's the best move. The study's key finding is that these two approaches don't agree with each other. In every condition tested, they produced near-disjoint vocabularies—meaning the "pieces" each method uses to represent molecules are almost entirely different from one another.
Alex: And why does that matter? If they both end up reading the same molecule, does it actually change anything?
Sam: It changes the model's entire "view" of chemistry. Think of it this way: when an AI model processes these pieces, it places each one in a kind of mathematical map—where pieces that are chemically similar end up close together, and dissimilar ones end up far apart. If you change the pieces, you change the map. Unigram-LM tends to cut molecules into many more, smaller fragments—up to about 40% more tokens than BPE for the same molecule. BPE builds chunkier, larger pieces. So the two models are essentially learning different languages to describe the same chemical structure.
Alex: So the "how" of tokenization isn't just a technical detail—it's a foundational choice that shapes everything downstream.
Sam: Precisely. The authors conclude that the choice of subword algorithm is a genuine modeling decision, not a free default you can ignore. And this divergence isn't a quirk of one particular dataset or one particular vocabulary size—it persists across every condition they tested. It's a structural difference baked into how these algorithms approach chemistry.
Alex: That raises a question for me. If you keep adding more and more pieces to the vocabulary, doesn't the model eventually start learning pieces it almost never sees? Like studying vocabulary words you'll never actually use?
Sam: That's a real concern, and the researchers looked at it directly. They measured something they call "clearance"—essentially, how often a learned piece actually shows up in the training data. What they found is that as the vocabulary grows larger, many pieces become what they call "undertrained." They appear so rarely that the model never gets enough examples to build a reliable understanding of what that piece means chemically.
Alex: So there's a natural ceiling on how useful a bigger vocabulary can be.
Sam: Right. And this connects to something specific about chemistry versus human language. When you write English, there are countless ways to combine words and phrases, so a large vocabulary of common chunks makes sense. But chemical formulas follow strict rules—there are only so many valid ways to arrange atoms. That means the space of high-frequency patterns is much more constrained. The "best" way to slice up a molecule isn't obvious, and the paper argues it's a question that deserves much more careful thought than it's currently getting.
Alex: So the broader implication is that researchers building chemical AI models may have been treating tokenization as a solved problem when it's actually still an open one.
Sam: That's the paper's central argument. By carefully isolating the tokenization algorithm from everything else—the dataset, the vocabulary size, the model architecture—the researchers show that this one choice alone produces fundamentally different internal representations. It's a call for more scrutiny at the very first step of model design, before any training even begins.
Alex: It's one of those findings that seems obvious in retrospect, but only once someone has actually done the work to demonstrate it clearly.
Sam: Exactly. The tools we borrow from one field don't always transfer cleanly to another. Chemistry has its own logic, its own constraints, and its own structure. The paper suggests that the field of chemical AI would benefit from building tokenization methods that are designed with that chemistry in mind, rather than inherited by default.
Alex: Thanks for walking us through that, Sam. And thanks to everyone listening to ResearchPod.