Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training

This paper introduces a novel method for synthetic data generation by "unfolding" existing scientific papers into multi-turn generation trajectories. Rather than rewriting text, the authors reconstruct the latent generative process behind 1.8 million arXiv papers. Each trajectory consists of a reconstructed writing request, a global plan, and pre-writing deliberations for each section, with the original section text and abstract kept verbatim. This process expands the corpus size by roughly 2x and provides models with a structured view of how complex documents are planned and executed.

The pipeline produces three distinct data assets from the same arXiv source:

• Continued Pre-training (CPT) Corpus: A 60B-token dataset of multi-turn trajectories that teaches models the rhetorical structure of scientific writing.
• Supervised Fine-tuning (SFT) Dataset: A 200K-sample dataset where real paper text serves as the answer, with tasks and thinking processes derived backwards.
• PAW-Bench: A new academic-writing benchmark consisting of 2,940 tasks anchored in held-out papers, each equipped with a specific grading rubric and a code-checkable checklist.

Controlled experiments demonstrate that continued pre-training on these trajectories provides a consistent performance boost across academic writing benchmarks compared to training on plain-text papers. Notably, this improvement persists even after subsequent supervised fine-tuning. Furthermore, the authors show that this specialized training does not harm general reasoning capabilities and actually improves performance on long-document reading tasks. The results suggest that the value of the synthetic data lies in the reconstructed structure of the writing process rather than the scale of the generator model used to create it.

As high-quality human text becomes scarce, synthetic data is essential for model scaling. This work demonstrates that reconstructing the "hidden thinking" behind long, structured documents is more effective for improving writing and reasoning than simply rewriting web text. By providing a scalable way to turn existing literature into training trajectories, the authors offer a robust recipe for enhancing the academic and long-context capabilities of large language models.