What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics

Outlier detection (OD) is a fundamental challenge in machine learning, particularly in unsupervised settings where the training data is contaminated with anomalies. A recent, empirically successful approach relies on the inlier-memorization (IM) effect: the observation that deep models, when trained on contaminated data, prioritize learning the patterns of normal data (inliers) before those of anomalies (outliers). This allows the reconstruction error of an under-fitted model to serve as an effective anomaly score. Despite its widespread use, the theoretical mechanisms governing why and when this effect occurs have remained largely unexplored.

This paper provides a rigorous theoretical foundation for the IM effect by analyzing a simple autoencoder trained via gradient descent. The authors show that the model naturally fits inliers earlier than outliers due to the underlying density and structural consistency of inlier clusters. The study characterizes the IM effect through two key metrics: its strength (the magnitude of the loss gap between inliers and outliers) and its persistence (the duration of the training window where this gap exists). The analysis reveals that the IM effect is most pronounced and durable when inliers form dense, compact clusters and when the initial model parameters are already aligned with the underlying data structure.

Building on these theoretical insights, the authors propose two actionable strategies to enhance the IM effect in real-world applications:

1. Compact Representations: Preprocessing data using embeddings from foundation models (e.g., ViT for images or TabPFN for tabular data) can project inputs into a space where inlier clusters are more compact, thereby reducing within-cluster variation and strengthening the IM effect.
2. EMA-based Initialization: Applying an exponential moving average (EMA) to model parameters during the early training phase helps suppress the influence of irregular outlier gradients, effectively guiding the model to capture the stable, inlier-dominated structure of the data.

These methods are simple to implement and require minimal changes to existing pipelines. Experiments across 57 datasets from the ADBench benchmark demonstrate that these strategies consistently improve the performance of existing IM-based methods, achieving state-of-the-art results.