Dataset Usage Inference without Shadow Models or Held-out Data

As large-scale generative models become central to legal and ethical disputes regarding data ownership, there is a critical need to determine not just whether a specific dataset was used for training, but to what extent. Existing methods for Dataset Usage Inference (DUI) are often impractical because they require training multiple 'shadow models' to mimic the target or demand access to a 'held-out' dataset known to be absent from training—resources rarely available to third-party auditors.

The authors propose NU-DUI, a framework that treats dataset usage inference as a Mixture Proportion Estimation (MPE) problem. To eliminate the need for shadow models and real held-out data, the authors:

1. Construct a synthetic non-member reference set by paraphrasing suspect images using a generative model from a different family than the target (e.g., using a diffusion model to paraphrase an autoregressive model's input).
2. Autoencode the suspect dataset using the same autoencoder used for paraphrasing, ensuring that both the suspect and synthetic sets share similar generative artifacts.
3. Extract membership inference (MIA) features from both sets.
4. Apply MPE to estimate the proportion of the suspect set that matches the training distribution.

NU-DUI provides accurate member-ratio estimates across five state-of-the-art image generative models, including both diffusion and autoregressive architectures. The method is highly efficient, achieving a speedup of over 2,000 times compared to traditional shadow-model-based approaches (e.g., 42.5 minutes versus 1,500+ hours on an A100 GPU). The authors demonstrate that by normalizing the suspect set through autoencoding, the estimation error remains low, effectively closing the performance gap between their practical approach and an 'oracle' setting that uses real held-out data.