Lethe: How Hard Is It to Forget? A Benchmark for Federated Unlearning in Medical Imaging

As medical imaging models are increasingly trained via federated learning (FL), the legal requirement to remove specific data—the 'right to be forgotten'—has become a critical challenge. Existing unlearning methods are primarily validated on natural images, which differ significantly from clinical data in heterogeneity and task structure. This paper introduces Lethe, a comprehensive benchmark designed to evaluate whether current federated unlearning methods effectively transfer to medical imaging tasks.

Lethe evaluates twelve representative unlearning methods across eight distinct task families, including classification, segmentation, denoising, and medical visual question answering. The benchmark employs a standardized protocol across three granularities of forgetting: client-level (a departing hospital), class-level (a retired label), and sample-level (a withdrawn patient). Performance is measured against a 'gold standard' retrained model using a seven-metric panel that assesses utility, privacy (via per-sample membership-inference attacks), cost, and durability.

The central discovery is that the choice of unlearning method is secondary to the difficulty of the request. On 'easy' requests—which dominate existing literature—most methods that preserve utility are indistinguishable from one another. Only 'hard' requests, such as class-level removal, effectively differentiate the methods. Furthermore, the researchers found that for many medical tasks, models generalize so effectively across sites that removing a client barely changes task performance. In these cases, the primary signal an unlearning method must erase is not the task contribution, but rather the residual membership of the data.