Georgi Ganev, Emiliano De Cristofaro
5 min
Synthetic data is increasingly adopted as a privacy-preserving alternative to sharing raw, sensitive datasets. However, current industry standards and regulatory assessments often treat synthetic data as a static, anonymized product, ignoring the underlying generative model. This paper argues that this dataset-centric view is insufficient. Because modern generative models (such as GANs, Diffusion Models, and LLMs) are often accessible via APIs or direct interaction, they must be treated as active components of the privacy risk profile. The authors contend that meaningful anonymity claims must be grounded in state-of-the-art privacy attacks that test the model's ability to memorize or reveal information about its training data.
The authors interpret the GDPR's requirements for anonymization—specifically the mitigation of singling out, linkability, and inference—through the lens of a 'motivated intruder.' They map these legal concepts to concrete, empirical privacy attacks:
By framing these risks as adversarial attacks, the authors provide a rigorous, testable methodology for assessing whether a synthetic data system meets the 'reasonably likely' threshold for anonymization required by regulators.
The paper compares two primary approaches to securing synthetic data: Similarity-based Privacy Metrics (SBPMs) and Differential Privacy (DP). The authors conclude that SBPMs, which measure the statistical distance between real and synthetic data, are inadequate for privacy protection because they do not account for the model's memorization capabilities and fail the motivated intruder test. In contrast, Differential Privacy provides a mathematically rigorous guarantee that limits the influence of any single individual on the model's output, offering a more reliable path toward regulatory compliance.
Training generative machine learning models to produce synthetic tabular data has become a popular approach for enhancing privacy in data sharing. As this typically involves processing sensitive personal information, releasing either the trained model or generated synthetic datasets can still pose privacy risks. Yet, recent research, commercial deployments, and privacy regulations like the General Data Protection Regulation (GDPR) largely assess anonymity at the level of an individual dataset. In this paper, we rethink anonymity claims about synthetic data from a model-centric perspective and argue that meaningful assessments must account for the capabilities and properties of the underlying generative model and be grounded in state-of-the-art privacy attacks. This perspective better reflects real-world products and deployments, where trained models are often readily accessible for interaction or querying. We interpret the GDPR's definitions of personal data and anonymization under such access assumptions to identify the types of identifiability risks that must be mitigated and map them to privacy attacks across different threat settings. We then argue that synthetic data techniques alone do not ensure sufficient anonymization. Finally, we compare the two mechanisms most commonly used alongside synthetic data -- Differential Privacy (DP) and Similarity-based Privacy Metrics (SBPMs) -- and argue that while DP can offer robust protections against identifiability risks, SBPMs lack adequate safeguards. Overall, our work connects regulatory notions of identifiability with model-centric privacy attacks, enabling more responsible and trustworthy regulatory assessment of synthetic data systems by researchers, practitioners, and policymakers.
Sam: Can you give me a concrete sense of what one of those tests looks like?
Alex: Sure. Take the risk of "singling out" a person—which is one of the GDPR's specific concerns. The researchers describe what's called a differencing attack. An attacker queries the model repeatedly and watches for unusually specific outputs. If the model keeps producing details that only match one particular person, that's a signal it has memorized a rare, individual record rather than learned a general pattern.
Sam: And there's a separate test for whether you can link a record back to someone?
Alex: Yes. That's called a membership inference attack. The attacker tries to determine whether a specific record was actually used during the model's training. If they can do that reliably, the model is leaking information about who was in the original dataset—which is a clear privacy violation under the law.
Sam: So the researchers are essentially building a bridge between legal definitions of privacy and concrete, testable attacks. That's a useful framing.
Alex: It is. And it leads directly to their recommendation on how to actually fix the problem. They compare two approaches. The first is what most companies currently use: ad-hoc similarity checks that look at whether the fake data resembles the real data too closely. The second is something called Differential Privacy.
Sam: What is Differential Privacy, in plain terms?
Alex: Imagine you're taking a class photo, but you want to protect one student's identity. Differential Privacy works by deliberately adding a small, carefully calculated amount of randomness—or "noise"—to the training process. The idea is that no single person's data should have a noticeable effect on what the model learns. If you removed one person from the training set entirely, the model's behavior shouldn't change in any detectable way. That mathematical guarantee is what makes it robust.
Sam: Whereas the similarity checks have no such guarantee—they're just eyeballing the output.
Alex: Exactly. The similarity checks can tell you the fake data doesn't look identical to the real data, but they can't tell you whether a determined attacker, querying the model through an API hundreds of times, could eventually reconstruct something private. Differential Privacy addresses the model's behavior at the source, during training, before any output is ever generated.
Sam: And that API point matters. In modern deployments, users interact with these models remotely—they can send query after query, refining their approach each time.
Alex: Which is precisely what makes the model-centric view so important. The paper's argument, at its core, is that the protection has to be built into how the model learns, not bolted on afterward by checking what it produces. If the model is the source of the risk, that's where the security has to live.
Sam: It's a shift from auditing the product to securing the process. That feels like a meaningful change in how the industry would need to think about this.
Alex: It does. And the stakes are real. The most severe scenario the paper describes is full reconstruction—an attacker actually recovering original, private records from the model. Until these systems are treated as potential repositories of sensitive information, and tested accordingly, the legal protections that privacy laws promise may not be backed by the technical reality.
Sam: That's a sobering place to land. Thanks for walking through it.
Alex: Thanks for listening to ResearchPod.