Leveraging AutoML for Sustainable Deep Learning: A Multi-Objective HPO Approach on Deep Shift Neural Networks

Research Question and Approach

Deep Learning models are increasingly resource-intensive, creating a need for more sustainable architectures. Deep Shift Neural Networks (DSNNs) address this by replacing computationally expensive multiplication operations with bitwise shift operations. However, DSNNs remain under-optimized. This paper investigates how to design more efficient DSNNs by using automated machine learning (AutoML) to navigate the trade-off between model accuracy and energy consumption.

The authors propose a multi-fidelity, multi-objective hyperparameter optimization (HPO) framework. By combining multi-fidelity techniques—which evaluate models on smaller budgets (e.g., fewer epochs) to save resources—with multi-objective optimization, the researchers explore a configuration space that includes shift depth, shift type (quantization vs. direct training), and bit precision. They use the ParEGO algorithm to scalarize the multi-objective problem, allowing them to identify Pareto optimal configurations that balance performance and emissions.

Key Findings

The study demonstrates that automated optimization significantly outperforms default DSNN configurations. By tailoring the hyperparameter search to the specific characteristics of shift operations, the authors identified configurations that simultaneously increase accuracy by about 20% while reducing energy emissions by up to 60%.

A notable finding concerns quantization strategy: contrary to common assumptions that uniform quantization is best, the researchers discovered that quantizing only smaller portions of the network with low precision can be optimal for energy savings while maintaining or even improving predictive performance. These results were consistent across multiple backbone architectures, suggesting that the automated approach is a robust tool for balancing efficiency and accuracy in resource-constrained environments.

As AI deployment shifts toward edge computing and IoT devices, energy efficiency is no longer just an environmental concern but a functional requirement. This research provides a practical, automated methodology for researchers to optimize DSNNs without manual, trial-and-error tuning. By offering a systematic way to explore the design space of shift-based networks, the authors enable the development of high-performance, low-power AI models suitable for real-time applications in constrained environments.