When Does Generating More Help? Disentangling Fixed-Source Synthesis from Source Expansion in Synthetic Data Scaling

Synthetic data is typically scaled by either enlarging the source (adding seed questions or changing the teacher) or by increasing the generation budget for a fixed source. While these two approaches are often conflated, this paper disentangles them into Source Expansion (SE) and Fixed-Source Synthesis (FSS). The authors investigate whether FSS provides a reliable scaling axis, how it compares to SE at matched total-sample budgets, and whether common synthesis protocols (like persona prompting or trace-level repair) actually improve performance when the source is held constant.

The authors isolate FSS by fixing the seed-question pool and the teacher model, then varying only the per-question response budget under Rejection Sampling (RS). They derive a three-parameter scaling law for FSS based on the coverage of latent features within the fixed source. To compare SE and FSS, they match the total number of training examples across different allocations of seed-question counts and response budgets. Finally, they audit various synthesis protocols—such as temperature tuning, embedding-based selection, and trace-level repair—by comparing them against a plain RS baseline under the same fixed-source control.

1. FSS follows a predictable, bounded scaling curve: performance improves with the response budget but exhibits diminishing returns as the student exhausts the signal available in the fixed source. The authors demonstrate that a parametric form fit on low-budget data accurately predicts performance at higher, held-out budgets.
2. SE is more reliable than FSS at large scales: while SE and FSS are comparable at small budgets, adding real seed questions (SE) consistently outperforms spending the same budget on more responses to existing seeds as the scale increases.
3. Synthesis protocols are largely ineffective under fixed-source control: once the seed pool and teacher are fixed, common techniques like persona prompting or trace-level repair do not consistently outperform plain RS. The apparent benefits of these methods in other studies are likely artifacts of budget effects rather than intrinsic improvements to the synthesis process.