Edge Physical AI Deployment of Vision Transformers on Heterogeneous Edge GPU Targeting Autonomous Vehicles

Modern autonomous vehicles rely on vision transformers for perception, yet these models often struggle with the strict latency and power constraints of edge AI-System-on-Chips (AI-SoCs). The primary challenge is that transformer-based architectures contain operators and tensor formats that are incompatible with fixed-function Deep Learning Accelerators (DLAs), forcing the system to fall back to the GPU. This fragmentation leads to underutilized hardware and inefficient power consumption. The authors propose H-FraDS, a hardware-aware scheduling framework that routes frames across the GPU, dual DLA cores, and an Optical Flow Accelerator (OFA) to improve utilization.

To enable DLA execution, the authors systematically identify and replace incompatible components in the Swin Transformer. First, they reshape 3D tensors into 4D formats to satisfy the DLA's memory-addressing requirements. Second, they approximate the Gaussian Error Linear Unit (GELU) activation, which relies on the error function (ERF), using a tanh approximation. Finally, they replace layer normalization with bounded tanh. These modifications allow the model to run as a continuous pipeline on the DLA, significantly reducing the frequency of GPU fallback transitions.

By implementing H-FraDS with a 1:2 dispatch ratio, the researchers achieved 125.93 FPS on the NVIDIA Jetson AGX Orin, comfortably exceeding the 30 FPS requirement for real-time autonomous driving. This configuration provided a 2.36x speedup over standalone DLA execution and maintained an efficiency of 4.0 FPS/W. Additionally, by integrating the OFA for inference-side optical flow estimation, the system achieved a 2.02x throughput speedup in the GPU-DLA-OFA configuration, demonstrating the benefits of concurrent heterogeneous hardware utilization.