Tzu-Heng Huang, Aditya Goyal, John Cooper, Frederic Sala
5 min
Foundation models are frequently released without disclosure of their training data recipes, specifically the domain mixture weights used during training. This creates an access asymmetry where researchers cannot audit the data composition of models they study. The authors investigate whether it is possible to reverse-engineer these domain proportions by analyzing only the released model weights, without access to the original training data or intermediate checkpoints.
The authors introduce WARP (Weight-space Analysis for Recovering Training Data Portfolios). Since the true training trajectory is typically unavailable, WARP uses model merging techniques (such as linear interpolation or TIES) to create a sequence of pseudo-checkpoints between the base model and the final fine-tuned model. By calculating the Mimic Score—a measure of how much a sample's gradient aligns with the direction of the fine-tuning path—the authors construct a geometric footprint of the training data. This footprint is then mapped to domain proportions using either a simple parameter-free softmax readout or a supervised MLP projector trained on synthetic mixtures.
WARP demonstrates that weight-space geometry contains a strong, recoverable signal regarding training data composition. In controlled experiments with BERT and GPT-2, WARP significantly outperformed sample-level membership inference baselines, achieving a mean absolute error (MAE) as low as 0.046 for BERT and 0.104 for GPT-2. Notably, the simulated trajectories created via model merging often outperformed the use of actual training checkpoints, likely because the simulated paths are smoother and less affected by the stochastic noise of real training processes.
This work provides a practical tool for researchers to audit foundation models, enabling better understanding of data contamination, capability drift, and the impact of data curation on model performance. By breaking the access asymmetry between model providers and the research community, WARP facilitates more transparent and reproducible AI research.
Foundation models are routinely released to the public, yet the data recipes used to train them -- such as domain mixture weights that determine how different sources are sampled -- are rarely disclosed. This creates an access asymmetry: researchers study the resulting models but lack visibility into the training distribution that produces them. Prior works for inferring training data, such as membership inference, detect at the level of individual samples and thus cannot characterize the global composition of the training corpus. We introduce WARP, a framework that recovers a fine-tuned model's training mixtures directly from its released weights. WARP interpolates between the base and fine-tuned models using model merging, generating pseudo-checkpoints that approximate the missing training trajectory and expose a geometric footprint of the training data in the weight space. From these simulated footprints, WARP extracts geometric features and maps them to domain proportions using either a parameter-free softmax readout or an MLP projector trained on synthetic mixtures. In controlled experiments with BERT and GPT-2, WARP recovers domain mixtures with an average MAE as low as 0.046 and 0.104 respectively, outperforming membership inference and a variant with access to the true training trajectory.
Alex: So you're measuring which types of data pulled the model the hardest — like checking which magnets had the strongest grip.
Sam: Exactly. And this is meaningfully different from older approaches. Previous methods, called membership inference, could only tell you whether one specific sentence was in the training data — one data point at a time. WARP produces what the researchers call a "geometric footprint" — a holistic view of the entire data mixture at once. In their experiments with models like BERT and GPT-2, this approach recovered training proportions with significantly higher accuracy than those older, sample-by-sample methods.
Alex: You mentioned something that caught my attention — that this simulated path actually outperformed an approach that had access to the real training logs. Why would a constructed path beat the genuine article?
Sam: It comes down to noise. Real training runs are messy. Data gets shuffled randomly, learning rates shift, and all sorts of unpredictable factors leave marks on the model's internal numbers. The smooth, linear path created by model merging filters out that randomness. The resulting footprint is cleaner, and a cleaner signal makes it easier to identify the underlying data mixture. The constructed path is, in a sense, a more idealised version of the truth — stripped of the clutter.
Alex: That's a genuine surprise. Are there limits to what this can do?
Sam: There's an important constraint. The method requires a representative sample of text from each domain you're trying to investigate. If you want to know whether a model was trained on medical data, you need some medical texts to test against — something to measure the alignment against. Without that starting material, the system has nothing to calculate. So it's an audit tool, but one that requires some prior knowledge of what you're looking for.
Alex: So it's not a universal key — it's more like a targeted probe. But if you have the right material, you can force a degree of transparency that the developers never intended to provide.
Sam: That's the practical implication. The authors suggest this could lead to automated data audits — where open-source models are routinely tagged with their training composition, making it possible to hold developers accountable for the data choices behind their systems.
Alex: There's something worth sitting with there. The geometry of numbers — the way values are arranged inside a model — ends up carrying a record of the human decisions that shaped it. What looks like a sealed black box turns out to be measurable.
Sam: That's the core insight. By treating a model as a physical object with a history, rather than just a tool that produces outputs, we can start to examine the layers of opacity that have defined much of recent AI development. And it suggests that secrecy around training data may be harder to maintain than many developers currently assume.
Alex: Thanks for listening to ResearchPod.