Hybrid Diffusion Transformer for Instruction-Guided Audio Editing via Rectified Flow

Instruction-guided audio editing requires modifying specific sound events within an audio clip based on natural language instructions while preserving the original acoustic content. Existing methods often struggle with either the computational cost of quadratic attention in transformer-based models or the limited semantic modeling capabilities of convolutional U-Net backbones. This paper explores how to design a more efficient and accurate diffusion transformer architecture that can handle complex instructions and overlapping sound events.

The authors introduce a hybrid two-stage diffusion transformer framework based on continuous-time rectified flow matching. The architecture employs a hierarchical design to manage computational complexity:

• Low-Resolution Stage: The model uses Dual-Stream Joint-Attention MMDiT (DSJA-MMDiT) blocks on downsampled audio tokens. This stage establishes coarse semantic alignment between the text instruction and the audio features.
• High-Resolution Stage: The model upsamples the features and processes them through alternating DSJA-MMDiT and AdaLN-Zero Cross-Attention DiT (AZCA-DiT) blocks. This stage focuses on refining local acoustic details and ensuring precise instruction following.

To improve controllability and content preservation, the model uses AdaLN-Zero modulation, which integrates global text, original audio, and timestep embeddings to guide the residual updates throughout the transformer blocks.

The proposed framework demonstrates significant improvements in both editing accuracy and inference efficiency. By separating the coarse semantic alignment from the fine-grained detail refinement, the model effectively handles challenging scenarios involving overlapping sound events and complex, multi-faceted instructions. The use of rectified flow matching further stabilizes the training process and reduces the number of inference steps required compared to traditional diffusion-based editors.