ResearchPod Summary
Large Language Model (LLM) pre-training is increasingly data-centric, with the composition of training data acting as a primary driver of model capability and training efficiency. While static data mixing strategies (using fixed ratios) are common, they fail to account for the non-stationary nature of learning dynamics. Existing online data mixing (ODM) methods often rely on single-objective optimizations that ignore the complex, multi-dimensional requirements of LLM training.
To address this, the authors propose the Holistic Data Scheduler (HDS), which treats data scheduling as a continuous control problem solved via the Soft Actor-Critic (SAC) reinforcement learning algorithm. HDS observes the model's state—including training loss, learning velocity, and weight norms—and dynamically adjusts the sampling probabilities for different data domains. The core innovation is a multi-objective reward function that balances three distinct signals: data quality, inter-domain gradient alignment, and model weight stability.
Empirical evaluations using a 1-billion-parameter Pythia model on The Pile dataset demonstrate that HDS significantly outperforms static baselines and previous ODM techniques. Key results include:
This research demonstrates that treating data composition as a dynamic, multi-objective control problem can substantially reduce the financial and environmental costs of training LLMs. By moving beyond static data recipes, HDS provides a scalable framework for optimizing the training process, allowing researchers to achieve higher performance with fewer compute resources. The inclusion of a stability-focused reward also offers a practical solution for preventing the training instabilities that often plague complex, large-scale model development.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.