An evaluation of the replicability of analyses using synthetic health data

As synthetic data becomes a standard privacy-preserving tool for sharing health data, it is critical to ensure that analyses performed on synthetic versions yield the same conclusions as those on real data. This study investigates how to maximize the replicability of logistic regression results using synthetic data, specifically examining the number of datasets to generate, the impact of data amplification, and the performance of different generative models.

The researchers conducted a simulation study using three heterogeneous real-world health datasets (colon cancer clinical trial data, Canadian community health survey data, and Danish colorectal cancer registry data). They evaluated two common generative models: sequential synthesis (using boosted decision trees) and a generative adversarial network (GAN). The study tested eight replicability metrics—including decision agreement, estimate agreement, and confidence interval coverage—across 320 scenarios, varying the number of synthetic datasets (m=1 to 20) and the level of data amplification.

The study demonstrates that using a single synthetic dataset without adjustment is often misleading and fails to provide valid population inferences. Instead, generating at least ten synthetic datasets and applying multiple imputation combining rules significantly improves replicability, ensuring that confidence intervals and statistical power align with real-world results. Sequential synthesis consistently outperformed the GAN across the datasets tested. Furthermore, while data amplification (creating synthetic datasets larger than the original) can marginally increase statistical power, it provides little additional benefit to overall replicability when proper combining rules are used. Membership disclosure risk remained low regardless of the number of datasets generated.