Sumyeong Ahn, Jongwoo Ko, Se-Young Yun
8 min
Deep learning models typically struggle with imbalanced datasets because they become biased toward majority classes. While previous techniques like re-weighting or re-sampling aim to fix this, they often fail to address the underlying lack of feature representation for minority classes. This paper investigates the interaction between the intensity of data augmentation and class-wise performance, uncovering a critical phenomenon: strongly augmenting majority classes improves the performance of minority classes, while augmenting a class too heavily can actually degrade its own performance. This insight motivates the need for a targeted, per-class augmentation curriculum.
The authors propose a Curriculum of Data Augmentation (CUDA) that automatically determines the appropriate level of augmentation for each class during training. CUDA operates using two primary mechanisms:
CUDA is designed to be plug-and-play, meaning it can be integrated into existing long-tailed recognition frameworks like CE-DRW, LDAM-DRW, or RIDE. Experimental results across benchmarks such as CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018 demonstrate that CUDA consistently outperforms baseline models. By balancing the weight norms of the linear classifier and improving the feature extractor's alignment, the method ensures that the network learns a more robust, balanced representation of all classes without requiring a separate validation set for tuning the augmentation policy.
Class imbalance problems frequently occur in real-world tasks, and conventional deep learning algorithms are well known for performance degradation on imbalanced training datasets. To mitigate this problem, many approaches have aimed to balance among given classes by re-weighting or re-sampling training samples. These re-balancing methods increase the impact of minority classes and reduce the influence of majority classes on the output of models. However, the extracted representations may be of poor quality owing to the limited number of minority samples. To handle this restriction, several methods have been developed that increase the representations of minority samples by leveraging the features of the majority samples. Despite extensive recent studies, no deep analysis has been conducted on determination of classes to be augmented and strength of augmentation has been conducted. In this study, we first investigate the correlation between the degree of augmentation and class-wise performance, and find that the proper degree of augmentation must be allocated for each class to mitigate class imbalance problems. Motivated by this finding, we propose a simple and efficient novel curriculum, which is designed to find the appropriate per-class strength of data augmentation, called CUDA: CUrriculum of Data Augmentation for long-tailed recognition. CUDA can simply be integrated into existing long-tailed recognition methods. We present the results of experiments showing that CUDA effectively achieves better generalization performance compared to the state-of-the-art method on various imbalanced datasets such as CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018.
Sam: Make “difficulty” concrete. What changes in an image when the strength goes up?
Alex: Strength controls both the number of image operations and their predefined magnitudes. The paper’s example uses strength three: brightness adjustment, horizontal shift, then vertical shift. The operations are randomly selected; their magnitudes depend on the strength setting.
Sam: That couples more operations with stronger operations. How does the model decide whether a particular class can handle the result?
Alex: Each class gets a Level-of-Learning score, meaning the augmentation difficulty the model can currently handle. At each training epoch, the method samples training examples from that class. It checks recognition at every strength from no added augmentation through the class’s current setting.
Sam: Why check the easier versions too? Passing only the hardest version might seem sufficient.
Alex: The update requires acceptable recognition across all those tested strengths, not just the current maximum. If every check passes the chosen threshold, the class’s strength rises one step. Otherwise, it falls one step.
Sam: So it’s a curriculum that can retreat, not a schedule that blindly gets harder. Does it need separate validation data to choose those class settings?
Alex: No separate validation set is needed for those updates. The checks use sampled training examples, with more samples tested at higher strengths. During training, the method also sometimes leaves images without the added CUDA transformations, to preserve access to their original information.
Sam: Let’s turn to the evidence. How large is the improvement when you compare the same underlying method with and without this curriculum?
Alex: On the severely imbalanced CIFAR benchmark, cross-entropy with deferred reweighting reaches about 41 percent accuracy. Adding CUDA takes it to about 48 percent. Deferred reweighting means introducing the balancing weights later in training.
Sam: That’s a substantial gain, but it’s also a particular benchmark and baseline. Did they repeat the comparison, and does it survive stronger starting methods?
Alex: The CIFAR results average three random trials. They also test multiple balancing approaches, an ensemble method, and contrastive learning, which trains useful feature relationships. The headline gains are not equally large across those starting points.
Sam: Give me a stronger-baseline comparison, so we don’t mistake the largest gain for the typical gain.
Alex: On ImageNet’s long-tailed benchmark, Balanced Contrastive Learning reaches 55.6 percent overall accuracy. Adding CUDA raises that to 56.3 percent. That’s a smaller improvement, but it shows the benefit isn’t restricted to a weak starting method.
Sam: Both benchmarks have deliberately constructed imbalance. Is there evidence from a dataset where the uneven distribution occurs naturally?
Alex: They also evaluate on iNaturalist, a large image dataset with naturally long-tailed categories. Adding CUDA improves overall accuracy across the underlying methods reported there. That broadens the evidence beyond synthetic subsampling, though it remains image classification.
Sam: What tells us the adaptive curriculum matters, rather than simply supplying more varied images?
Alex: Several comparisons address that. Using one shared strength for every class performs worse than class-specific adaptation. Retraining from scratch with CUDA’s final class strengths also performs worse than learning through the curriculum. And applying many random augmentations can perform worse than the baseline.
Sam: Those comparisons support both parts of the proposal: separate settings for separate classes, and a changing schedule. What explains the effect on classes that weren’t augmented?
Alex: The authors examine classifier weight norms: the sizes of the learned weights for each class. In their analysis, augmented classes develop more varied training features and smaller classifier weight norms. That can reduce their output scores relative to unaugmented classes.
Sam: So the proposed explanation involves changing the balance between classes, not just improving recognition of altered images. Do they also see better features?
Alex: Their validation analysis finds improved within-class feature alignment for almost all classes. That means examples from the same class have more similar feature directions. They also find less variation in classifier weight norms across classes, which they interpret as a more balanced classifier.
Sam: What should stop a reader from treating this as a general solution to uneven training data?
Alex: The tests concern image classification, with imbalanced training and balanced test distributions in the stated setup. Overall gains don’t guarantee every class group improves; some reported subgroup results decline. And adaptation has a cost: one Balanced Softmax example takes about 29 percent more training time.
Sam: Given that scope and cost, who should read the full document, and where should they start?
Alex: Researchers training image classifiers with scarce categories should read it, especially if they already use a balancing method. Start with the curriculum description in Section Three Point Two. Then read the analysis section and appendix comparisons that remove class-specific adaptation or freeze the final strengths.
Sam: And what should everyone else take away, without carrying the implementation details around?
Alex: Augmentation is not only about the class you change. Making a well-learned class harder can help another class—but the model needs a way to say when that’s too hard.
Sam: Thanks for listening.