Memory Efficient Audio Synthesis with Decoupled Temporal Depth Diffusion Transformers

As on-device foundation models like Apple's AFM 3 Core Advanced grow in capability, they require efficient ways to convert abstract semantic audio tokens into high-fidelity, real-time speech. Traditional transformer-based audio synthesis often suffers from linear or quadratic memory scaling, making them unsuitable for resource-constrained mobile environments. This paper investigates how to design a memory-efficient detokenizer that can sustain continuous streaming audio synthesis within a strict, shared on-device memory budget.

The authors propose a three-component architecture: a streaming encoder, a temporal decoder, and a unified depth decoder. The key innovation is the systematic decoupling of temporal and depth processing. Unlike prior multi-decoder architectures that require dedicated parameters for each residual vector quantization (RVQ) level, this model uses a single, reusable depth decoder. This decoder identifies specific RVQ levels using Diffusion Transformer (DiT)-style stage conditioning—specifically, rotary positional encoding of the codebook index. To ensure memory efficiency, the system employs causal sliding window attention with fixed-window key-value caching, which decouples memory usage from the total length of the audio sequence.

The proposed architecture demonstrates significant computational advantages over existing methods. In production deployment on the Apple Matrix Coprocessor (AMX), the detokenizer achieves a generation speed of 16x faster than real-time, with a peak runtime memory of approximately 21 MB. This constant memory footprint allows for continuous streaming of audio sequences ranging from 20 to 320 seconds. Comparative benchmarks on a common GPU platform confirm that while baseline transformer-decoder and GAN-based models see their memory and generation time scale with sequence length, the proposed model maintains constant performance. Furthermore, the architecture improves the Mean Opinion Score (MOS) for conversational speech by +0.42 compared to previous on-device systems.