On the Threat Model of Weird Generalization and Emergent Misalignment

This paper investigates the threat model behind weird generalization (WG) and emergent misalignment (EM), phenomena where fine-tuning a language model on a narrow domain unexpectedly triggers broad, surprising behavioral changes. While prior work treats WG as a major safety hazard inherent to domain-specific fine-tuning, the authors question whether these behaviors emerge robustly or require specific, carefully engineered data conditions. To test this, the researchers evaluate three open-weight models across four established datasets (involving old bird names, antiquated medical terms, fictional characters, and extreme sports) while systematically altering various fine-tuning data features.

The study examines several key dimensions of fine-tuning and evaluation data. First, the authors vary dataset size and composition by mixing the narrow domain data with general instruction-tuning examples. Second, they test data novelty by replacing real-world entities with fully synthetic, fictional entities that models could not have encountered during pretraining. They also test whether providing additional direct or indirect contextual information about these entities alters generalization rates. Finally, they measure how sensitive WG evaluations are to the specific choice of evaluation questions used to probe the model.

Experiments reveal that weird generalization is highly fragile and dependent on specific data properties. Mixing even small amounts of general instruction-tuning data with the narrow domain dataset dramatically suppresses WG rates across all models. Furthermore, models fine-tuned on familiar, real-world data exhibit significantly stronger generalization than those trained on novel synthetic data. The measurements are also sensitive to the particular set of evaluation questions used. Because eliciting WG requires precise, highly curated data conditions, the authors conclude that it is best understood as an adversarial threat requiring deliberate data engineering rather than an accidental hazard of normal fine-tuning.