Scaling Audio Models Efficiently: A Joint Study of Compute Constraints and Optimization Behavior

This paper investigates how to optimally allocate computational resources—measured in floating-point operations (FLOPs)—across three specific dimensions: model size (xN), input length (xT), and representation resolution (xV). The authors aim to maximize performance for two distinct audio tasks: Automatic Speech Recognition (ASR) and Speech Emotion Recognition (SER). By systematically varying these axes under fixed compute budgets, the researchers seek to identify Pareto-optimal configurations that provide the best performance-to-cost ratio for real-world deployment.

The study reveals that scaling behavior is highly task-dependent. For ASR, the Pareto frontier is smooth and monotonic; increasing model size yields diminishing returns, while reducing encoder token resolution (xV) serves as a highly effective mechanism to lower inference costs with minimal impact on Word Error Rate (WER). For instance, reducing token resolution in the Large-v3 model cut inference costs by over 50% with less than a 3% relative increase in error.

In contrast, SER exhibits a sparse Pareto frontier. The authors found that SER performance is sensitive to audio duration, with an optimal clip length of approximately 4 seconds; shorter clips lack sufficient prosodic context, while longer clips introduce noise. Furthermore, standard LoRA-based adaptation was insufficient for SER, requiring the use of DAMA (Depth-Aware Model Adaptation) to selectively unfreeze upper encoder layers to achieve competitive accuracy.

As large audio models become standard, their high computational requirements often preclude deployment on edge devices. This research provides a principled framework for engineers to navigate the trade-offs between model capacity and inference latency. By identifying that compute axes like token resolution and input length are often more efficient levers than simply scaling model parameters, the authors offer a roadmap for building more sustainable and efficient speech-processing systems.