Filling the Pareto optimal front for affordance segmentation on embedded devices using RGB-D cameras

As wearable robots increasingly require semi-autonomous control, the ability to perform affordance segmentation (AS)—identifying functional parts of objects—on resource-constrained embedded devices is critical. While depth (D) sensors provide valuable geometric information, their integration into lightweight deep neural networks (DNNs) remains underexplored due to the strict hardware constraints of portable platforms. This paper investigates how to effectively incorporate RGB-D data into efficient DNN architectures without exceeding the energy and computational budgets of embedded systems.

The authors propose two distinct strategies to balance accuracy with hardware efficiency:

1. Hardware-Aware Neural Architecture Search (HW-NAS): A custom search space is designed to optimize the structure of multi-branch networks that process RGB and depth inputs independently before merging them. This approach uses an evolutionary algorithm to find architectures that satisfy specific inference time and memory constraints.
2. Fine-Tuning with Preprocessing: A more lightweight alternative that takes a pre-trained RGB-only network and adds a learned preprocessing layer. This layer maps the 4-channel RGB-D input into a 3-channel representation compatible with existing architectures, allowing for rapid adaptation with minimal additional computational cost.

Extensive testing on the UMD and IIT datasets demonstrates that both proposed methods outperform existing hardware-efficient solutions. The resulting models consistently lie on the Pareto optimal front, providing the best possible accuracy for a given computational budget (FLOPs) and model size. The authors successfully deployed these models on a Jetson Nano prototype paired with a RealSense camera, confirming that the system can achieve real-time inference within an energy budget suitable for battery-powered wearable devices.