MUSIC RESTORATION VIA LATENT OPERATOR OPTIMIZATION AND DIFFUSION MODEL PRIORS

Music restoration—the process of recovering a clean, dry signal from a recording processed by unknown effects or distortions—is typically handled by supervised models trained on specific, known degradation pipelines. This paper addresses the challenge of performing restoration when the distortion process is unknown, complex, or lacks paired training data, aiming to create a general-purpose, unsupervised framework.

The authors introduce LOUDAR (Latent-space Optimization of Unknown Distortion for Audio Restoration). Instead of working directly on the waveform, LOUDAR operates in the latent space of a pretrained audio autoencoder. The core of the method is an iterative inference procedure that alternates between two main steps: estimating the clean latent representation using an unconditional latent diffusion model (which acts as a prior for clean audio) and updating a learnable latent operator that models the unknown distortion. This Expectation-Maximization (EM) style approach allows the model to adapt to the specific degradation of each input recording at test time.

LOUDAR demonstrates consistent improvements over degraded inputs across diverse tasks, including singing voice effect removal and guitar distortion removal. The authors show that their approach is competitive with both supervised and unsupervised baselines. By parameterizing the distortion as a compact, causal convolutional network in latent space, the model effectively balances the flexibility needed to handle complex effects with the constraints required to prevent overfitting during the blind optimization process.

This work provides a flexible solution for audio restoration that does not require paired training data or prior knowledge of the distortion chain. It is particularly useful for music production, remixing, and preparing high-quality training data for other audio technology systems, where the original, unprocessed source material is often unavailable.