Spaghetti Architect: A Contamination-Resistant, By-Construction-Labelled, Multi-Language Code Dataset Generator

Traditional code corpora are mined from the wild, making them uncontrolled, prone to contamination, and lacking in precise difficulty labels. Researchers often struggle to isolate whether a model fails because a coding problem is inherently difficult or because the code's presentation is unnecessarily messy. Spaghetti Architect addresses this by providing a tool that generates code datasets with strict control over semantics, complexity, and contamination.

Spaghetti Architect functions as an anti-optimization transpiler. It takes a clean, language-agnostic JSON intermediate representation (IR) and transforms it into redundant, fully-flattened code in five languages (Python, JavaScript, Go, Java, and C++). The tool uses a 'messiness' knob—a series of strictly-nested anti-pattern profiles—to increase incidental complexity while keeping the underlying semantics identical. Every generated program is compiled and verified against a reference oracle to ensure correctness by construction. The tool also implements a contamination-resistance protocol, allowing users to mint fresh, private test splits from a held-out seed, ensuring that evaluation data remains unseen by models during training.

The study demonstrates that the dataset is both usable and discriminative across a four-model capability ladder. Key results include:

• Orthogonal Difficulty: The generator successfully separates intrinsic problem size from incidental presentation, allowing researchers to isolate the causes of model failure.
• Contamination Resistance: Scores on public development sets closely match those on freshly minted, private test sets, confirming that the protocol effectively mitigates memorization.
• Competence Dissociation: The dataset reveals that while output-prediction accuracy collapses as intrinsic problem size increases, refactoring equivalence remains invariant for capable models, effectively separating computational tasks from structural ones.
• Annotation Inflation: The generator's self-annotations provide hints that inflate model performance, with a significantly larger impact on weaker models, which can artificially compress the performance gap between models.