Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining

This paper investigates the phenomenon of "natural ungrokking," where a language model acquires a linguistic rule during pretraining only to lose it later in the same run, despite the rule's evidence remaining present in the training data. The authors examine whether this loss is a result of the model forgetting the construction entirely or a displacement where the model retains the construction but favors a competing surface pattern. To test this, the researchers used a 11.5M-parameter transformer model trained on two distinct corpora—TinyStories and a filtered web corpus—and tracked performance using a frozen battery of conflict probes and a contrast margin metric.

The study establishes that a rule's survival is determined by its "support frequency"—how often the training stream shows the rule winning. While the ratio of data to parameters modulates the depth of the collapse, it does not determine the final outcome. The researchers found that the loss is a displacement: as the rule collapses, the log-probability margin between the rule-conforming continuation and the competing surface prior crosses zero within 100 training steps of the behavioral collapse. Furthermore, the authors demonstrate an asymmetric control: while they could easily destroy rules by injecting counter-evidence, they could not restore collapsed rules even by injecting support at 450 times the natural level. These dynamics were also observed in larger public models, such as Pythia and OLMo, where the depth of collapse was ordered by model scale.

This research provides a mechanistic understanding of why models "forget" or fail to generalize certain rules during training. By showing that rule fate is predictable and controllable through corpus statistics, the authors offer a framework for understanding training dynamics that are otherwise invisible to standard loss curves. This suggests that the quality and frequency of evidence in a training corpus are critical for ensuring that learned capabilities persist through the end of pretraining.