SCICODEPILE: A 128GB Corpus and Executable Benchmark for Challenging Scientific Code Generation

Large language models (LLMs) have demonstrated impressive general-purpose coding abilities, but their performance in specialized scientific domains—such as quantum chemistry, bioinformatics, and molecular docking—remains poorly understood. Scientific code requires adherence to complex numerical logic, domain-specific semantics, and project-level dependencies that general benchmarks often fail to capture. Existing datasets are typically either too small for effective training or lack the executable verification necessary to distinguish between syntactically plausible code and functionally correct scientific implementations.

The authors introduce SciCodePile, a comprehensive resource designed to bridge this gap. The corpus is constructed from 37,737 curated public repositories, totaling 128GB of scientific code. The construction pipeline employs a retrieve-then-filter strategy, using LLM-assisted vocabulary expansion and relevance checks to ensure high-quality data.

SciCodePile provides data in four distinct formats:

• F1 (Raw Code): A large-scale collection for continued pretraining.
• F2 (README Summary): Structured repository-level context.
• F3 (Function Instruction): Function-level code paired with natural-language descriptions.
• F4 (Problem–Solution Pairs): Instruction-style training signals for model fine-tuning.

Additionally, the authors curated an executable benchmark of 200 tasks. Each task includes a sandboxed environment and an automated test harness, ensuring that model performance is measured by functional correctness rather than simple text similarity.

Evaluating 15 diverse LLMs, the authors found that scientific code generation remains highly challenging. Even the strongest models achieved only 12.30% Pass@1 on the executable benchmark, highlighting a major discrepancy between general coding proficiency and scientific domain expertise. However, the study confirms the utility of the dataset: continued pretraining on SciCodePile improved CodeBLEU scores by 2.84x on completion tasks, and instruction tuning on the F3/F4 subsets increased Pass@1 performance by 4.79x, demonstrating that the corpus provides a high-quality training signal for scientific domains.