Auditing Data Provenance in LLM Fine-tuning via Intrinsic Distributional Fingerprints

As Large Language Models (LLMs) are increasingly fine-tuned on proprietary, high-value datasets, Data Owners face a significant risk of intellectual property infringement. Traditional auditing methods, such as watermarking or membership inference attacks, are often ineffective because they either require pre-training intervention or fail to distinguish between proprietary data usage and synthetic decoys when an adversary employs obfuscation techniques like paraphrasing or knowledge distillation. This paper addresses the challenge of verifying data provenance in a black-box, post-hoc setting.

The authors propose the Distribution Provenance Audit (DPA), a framework that shifts the focus from record-level detection to distribution-level alignment. The core insight is that for an LLM to maintain expert-level utility, it must internalize the intrinsic statistics of its training data. DPA captures these as 'distributional fingerprints'—the intersection of semantic substance and lexical form. The framework utilizes two key components: Joint Semantic-Lexical Similarity (JSLS) to quantify these fingerprints, and Self-Instruct Data Extraction (SIDE) to faithfully reconstruct the model's latent distribution without prompt-induced bias. The audit is then formulated as a statistical hypothesis test to determine if a model's output distribution aligns with the proprietary source.

Experimental results on medical and legal datasets demonstrate that DPA significantly outperforms existing baseline methods. Even when malicious trainers attempt to hide data usage through subset splitting, paraphrasing, or knowledge distillation, DPA successfully identifies the proprietary fingerprints with high statistical confidence. The authors also identify a 'dual-use tension': the same high-fidelity fingerprints that allow for reliable provenance auditing could theoretically be exploited to facilitate privacy attacks, highlighting a trade-off between model transparency and data security.