Understanding Machine Unlearning Through the Lens of Mode Connectivity

Research Overview

This paper introduces Mode Connectivity in Unlearning (MCU), a framework designed to analyze the loss landscape and optimization geometry of machine unlearning. While standard mode connectivity explores paths between independently trained models, MCU investigates whether unlearned models—which must simultaneously minimize retain-set loss and maximize forget-set loss—can be connected by smooth, low-loss paths in parameter space. The authors evaluate this across various language model and classification benchmarks, including curriculum learning and second-order optimization settings.

Key Findings

The researchers demonstrate that unlearned models often reside within connected, low-loss basins. A critical discovery is that these basins are "functionally heterogeneous"; while models within the same basin may show similar loss values, they can differ drastically in their performance on specific metrics, particularly regarding privacy leakage and utility retention. Furthermore, the study reveals that unlearning is a non-linear process: models often maintain original performance for a significant portion of the parameter path before rapidly committing to forgetting. The authors also find that MCU smoothness serves as a proxy for unlearning difficulty, and that MCU-based ensembling can enhance both model generalization and robustness against relearning attacks.

Mechanistic Insights

By examining linear and quadratic paths between unlearned models, the authors conclude that most approximate unlearning methods are mechanistically distinct from full retraining. The study also highlights that changing training dynamics, such as the introduction of curriculum learning or second-order optimization, can shift solutions into entirely different basins. This suggests that the choice of optimization strategy fundamentally alters the geometry of the resulting unlearned model, providing a new diagnostic tool for researchers to evaluate and compare different unlearning algorithms.

Understanding the loss landscape of unlearning is essential for building trustworthy AI. By showing that unlearning is not a uniform process and that models can be connected in parameter space, this work provides a rigorous way to evaluate the stability and reliability of unlearning methods. The findings suggest that current evaluation protocols, which often rely on aggregate metrics, may miss critical variations in model behavior, and that MCU offers a promising path toward more robust model selection and improved unlearning performance.