Speaker: Kumar Kshitij Patel, Yale Institute for Foundations of Data Science (FDS)
Abstract: Diffusion models are now the dominant approach for high-fidelity image generation, yet they remain highly vulnerable to privacy attacks, including reconstruction and membership inference attacks (e.g., Carlini et al.), which limit their deployment in sensitive domains such as healthcare and finance. Existing approaches based on differentially private training often substantially degrade sample quality, weakening the very capabilities that make diffusion models effective. In this talk, I will present a framework for training diffusion models in federated settings with heterogeneous client data. Our approach combines two key ideas: personalization to reduce the tension between privacy and utility, and the coarse-to-fine refinement structure inherent in diffusion models. Specifically, a shared global model learns the coarse structure that generalizes across clients, while client-specific models perform the finer refinements that encode local information. This decomposition allows clients to benefit from collaboration while limiting what the shared model can memorize, since it only accesses noisy privatized versions of each client’s data.
Our framework provides formal local differential privacy guarantees at the image level while preserving much of the fidelity that makes diffusion models effective. We further show, in a Gaussian mixture model, that collaboration in our framework provably improves sample quality compared with fully local training. Experiments on standard image datasets support these findings: the method produces high-fidelity samples, improves generation quality for minority and underrepresented classes, and maintains strong protection against membership inference, memorization, and reconstruction attacks. The talk is based on our CVPR 2026 paper.
This will be a free online seminar. To register to attend, please click here: https://cam-ac-uk.zoom.us/meeting/register/09COy6SHTNqiWXLEC1ouwQ