Internal Data Repetition Destroys Language Models

As high-quality training data becomes scarce, practitioners are increasingly forced to reuse data. This study investigates the impact of exact document-level repetition on language model performance. Specifically, the authors seek to quantify the "compute-equivalent" cost of repetition—how much training compute is effectively wasted when a portion of the training data is repeated.

The researchers trained a series of Qwen3-style decoder-only transformers with parameter counts ranging from 34M to 344M. They maintained a fixed 10% repeated-token fraction while varying the "repetition structure" (the number of times a specific subset of documents is replayed). By comparing these runs against a no-repetition baseline using a fitted Chinchilla-style scaling law, they calculated the Compute-Equivalent Gain (CEG) and Compute-Equivalent Loss (CEL). This allowed them to map performance degradation directly to wasted FLOPs.

The study reveals that repetition damage is non-monotonic: performance does not simply worsen as more data is repeated. Instead, evaluation loss peaks at an intermediate repeat count, suggesting a specific "worst-case" configuration where the model struggles to balance memorization of the repeated pool with generalization to the rest of the corpus. This peak location follows a predictable power law relative to model size. At the largest scale tested (344M parameters), the most damaging repetition structure resulted in a 33% compute-equivalent loss, demonstrating that even moderate amounts of repetition can significantly undermine training efficiency.

These findings provide a precise, quantitative framework for understanding data quality in the era of data-constrained scaling. By showing that repetition damage is not just a model-specific quirk but can be modeled as a statistical tradeoff, the authors offer a tool for practitioners to estimate the potential compute waste in their pretraining pipelines. This suggests that aggressive deduplication is not just about cleaning data, but is a critical lever for maximizing the return on massive compute investments.