Neural Architecture Search of Sample Reweighted Networks for Complex Distribution Shift

Sample reweighting is a standard technique for handling distribution shifts like label noise and class imbalance. While Meta-Weight-Net (MW-Net) effectively addresses these issues individually using a simple neural network, its performance drops when both shifts occur simultaneously. The authors investigate whether optimizing the architecture of the weighting network and the specific features it uses as input can improve robustness in these complex, multi-factor shift scenarios.

The authors propose two primary modifications to the standard MW-Net framework. First, they expand the input to the weighting network beyond just the classification loss, incorporating intermediate feature representations from the classifier and one-hot encoded label information. Second, they apply Neural Architecture Search (NAS) using the Tree-structured Parzen Estimator (TPE) to automatically determine the optimal depth, number of nodes, and the specific intermediate layer of the classifier to use as input. This allows the weighting network to adapt its complexity and feature source to the specific demands of the dataset.

Experimental results on CIFAR-10 and CIFAR-100, modified to include both label noise and class imbalance, demonstrate that the proposed NAS-based approach consistently outperforms the original MW-Net architecture. The study shows that the baseline model struggles to distinguish between noisy samples and minority class samples, both of which often exhibit high loss. By contrast, the optimized weighting network successfully assigns lower weights to noisy samples while maintaining higher weights for minority class samples, leading to improved classification accuracy across various noise and imbalance levels.