Scaling LLM Knowledge Boundaries via Distribution-Optimized Synthesis

How can we improve the efficiency of knowledge injection in Large Language Models (LLMs) using synthetic data? The authors argue that current methods suffer from a "blind-synthesis" paradigm, where data is generated without regard for its underlying knowledge distribution, leading to redundant information in some areas and critical sparsity in others.

The researchers propose KDoS (Knowledge Distribution-optimized Synthesis), a framework that treats knowledge injection as a distribution control problem. KDoS introduces "knowledge density" as a metric to quantify how knowledge is distributed in semantic space. The framework operates in three stages: extracting and grouping knowledge from seed data, filtering for quality, and using a rejection sampling mechanism to dynamically adjust the data pool toward a target density. This allows the model to move away from arbitrary token counts and toward a distribution that maximizes learning efficiency.

• Optimal Distribution: There is a consistent, optimal knowledge density range (approximately 10⁻⁴ to 10⁴ in their experiments) that maximizes the expansion of an LLM's knowledge boundary.
• Stability: This optimal distribution is stable across different model backbones (e.g., Qwen, LLaMA, Ling) and scales, suggesting a universal principle for synthetic data-driven injection.
• Performance: KDoS consistently outperforms standard baselines (random, uniform, and difficulty-weighted synthesis) across six knowledge-intensive and long-tail benchmarks, including TriviaQA and SimpleQA.
• Knowledge Collapse: The study identifies a "Knowledge Collapse Region" at excessively high densities where model performance degrades sharply, indicating that over-concentrating knowledge is detrimental to learning.

This work shifts the focus of synthetic data generation from simple volume scaling to distribution optimization. By providing a practical framework to control knowledge density, researchers can more effectively target long-tail knowledge gaps, which are notoriously difficult for LLMs to master. This approach offers a more systematic way to scale model capabilities without the diminishing returns associated with blind data accumulation.