Low-Burden Data Augmentation for Dysarthric ASR via Zero-Shot Voice Cloning

Automatic speech recognition (ASR) systems often struggle with dysarthric speech due to high inter-speaker variability and severe data scarcity. Collecting large, high-quality datasets from individuals with dysarthria is logistically difficult and expensive. This paper investigates whether zero-shot voice cloning—using a foundation model to synthesize speech from a single reference utterance—can serve as an effective, low-burden data augmentation strategy to improve ASR robustness without requiring extensive new recordings.

The researchers used the Higgs Audio V2 model to clone eight speakers from the TORGO dataset, using only a single 7.2-second reference utterance per speaker. They generated a synthetic dataset (TORGO-Synth) consisting of 15 hours of speech using linguistically diverse text prompts. They then fine-tuned the Whisper-medium ASR model on three configurations: real speech, synthetic cloned speech, and a hybrid of both. The performance was evaluated on held-out real speech from TORGO and on a cross-corpus test set from the Speech Accessibility Project (SAP-1102).

Fine-tuning on cloned speech significantly reduced the Word Error Rate (WER) from 31.62% (zero-shot baseline) to 26.00%. While fine-tuning on real data remained the most effective overall, the synthetic clones proved superior for moderate-to-severe speakers, where the model benefited from the increased phonetic diversity. The study also identified an optimal 'sweet spot' of 15 hours of synthetic data; exceeding this volume led to diminishing returns and potential overfitting to synthesis artifacts. Furthermore, the clone-based model showed better cross-corpus generalization than the real-data model, suggesting that synthetic augmentation helps bridge domain gaps between different dysarthric datasets.