Wenlong Cheng, Yuan Gan, Yunqiu Xu, Jiaxu Miao
4 min
Federated Learning (FL) allows multiple participants to train Latent Diffusion Models (LDMs) without sharing raw data. However, this decentralized process exposes the global model to malicious clients who may distribute or resell it. Existing watermarking techniques for LDMs are either limited to ownership verification or are vulnerable to simple VAE decoder replacement attacks, where a malicious user swaps the watermarked decoder for a clean one to erase the watermark without degrading image quality.
To address these issues, the authors propose FedOT. The framework introduces two primary innovations: a chunked watermark mechanism and Latent Vector Transformation (LVT). The chunked watermark splits the binary watermark into two parts: a prefix for verifying that the model belongs to the federated group, and a suffix unique to each client for tracing the source of a leak. To prevent VAE replacement, LVT modifies the latent space distribution of the VAE before federated training begins. This forces the U-Net to adapt to a non-standard latent distribution, ensuring that the model components are tightly bound; replacing the VAE with a standard version results in significant image degradation.
The authors evaluated three LVT strategies—translation, mirror, and negative transformation—to determine the best balance between component binding strength and generative fidelity. They found that the negative transformation, which implicitly reshapes the latent distribution by training the VAE to map inputs to their pixel-inverted counterparts, provides the most robust protection. Experiments demonstrate that FedOT successfully verifies ownership and identifies the specific malicious client responsible for a leak. Furthermore, the LVT mechanism effectively deters VAE replacement attacks, as any attempt to swap the VAE decoder renders the LDM unusable due to severe image quality loss.
As generative models become increasingly valuable, protecting intellectual property in collaborative training environments is critical. FedOT provides a practical, server-side solution that secures federated LDMs against both unauthorized distribution and malicious tampering, ensuring that model providers can maintain accountability in distributed AI development.
Training Latent Diffusion Models (LDMs) within Federated Learning (FL) has attracted increasing attention due to its ability to combine the powerful generative capacity of LDMs with the privacy-preserving properties of FL. However, FL requires sharing the global model with multiple participants, which risks unauthorized model distribution or resale by malicious clients. While an intuitive approach is to adopt existing VAE-based watermarking techniques for LDMs in FL, this strategy falls short in addressing such threats due to two fundamental challenges: (1) Existing methods support ownership verification but lack the ability to trace model leakage to a specific malicious client; (2) VAE-based watermarks are vulnerable, as they can be removed simply by replacing the decoder with a clean counterpart. In this paper, we propose FedOT, the first framework for ownership verification and leakage tracing in federated LDMs. Specifically, to address the first challenge, we design a chunked watermark, where the first part is for ownership verification, and the second part is used for client identification. Furthermore, to overcome the second challenge and secure the model against VAE replacement attack, we introduce Latent Vector Transformation (LVT), which strengthens the connection between the VAE and U-Net latent spaces by modifying the original latent distribution of the VAE. Consequently, any attempt to replace the VAE for watermark removal leads to significant image quality degradation, making the LDM model unusable. Extensive experiments demonstrate that FedOT achieves superior performance in both ownership verification and traceability. Project page: https://spyzixuan.github.io/FedOT/.
Sam: Okay, but here's a follow-up. If I'm the model owner and I have ten partner organisations, and one of them leaks the model—how do I know which one it was?
Alex: The paper addresses that with what they call a "chunked watermark." The ownership signal is split into two distinct parts. The first part is shared across all copies and simply confirms that the model belongs to your organisation. The second part is a unique code—different for each partner—that acts like a serial number on that specific copy.
Sam: So it's a two-step check. First you confirm it's your model at all, and then you look at the serial number to identify who was holding that particular copy.
Alex: Exactly. And it's efficient—you only bother checking the serial number once you've already confirmed the model is yours. It avoids a lot of unnecessary searching.
Sam: There has to be a tension here, though. If you're deliberately distorting how the model processes information, doesn't that hurt the model's quality for the honest users who are just trying to use it legitimately?
Alex: That is the genuine trade-off the researchers had to navigate. The transformation has to be subtle enough that legitimate users see no meaningful drop in quality, but significant enough that removing it causes the model to fail. The paper describes this as a careful balancing act—and the researchers acknowledge it as the central design challenge. They don't claim it's fully solved, but they argue their approach manages the balance more effectively than previous methods.
Sam: So in a sense, they've turned the model's dependency into its own defence. The security isn't protecting the model from the outside—it's built into the model's core logic.
Alex: That's a good way to put it. The insight is that a watermark you can remove is always vulnerable. But if the security layer is the foundation, then removing it doesn't free the model—it breaks it. Whether that proves durable against more sophisticated attacks remains to be seen, but as a design principle, it's a meaningful shift in how people are thinking about AI model protection. Thanks for listening to ResearchPod.