Robust Assamese Speech Recognition through Controlled Fine-Tuning of Whisper Models

Automatic Speech Recognition (ASR) for low-resource languages like Assamese is hindered by a lack of annotated data and the tendency of large, pretrained models to hallucinate or perform poorly on specific linguistic features. The researchers aimed to improve transcription accuracy for Assamese by fine-tuning the Whisper-Small model. They utilized the Mozilla Common Voice 24.0-Assamese corpus, augmenting the official validated training set with unvalidated crowd-sourced clips to increase the total training data. To accommodate resource-constrained environments, the team implemented a hardware-aware pipeline using mixed-precision training and gradient accumulation on T4 GPUs.

The fine-tuned model demonstrated substantial performance gains compared to the zero-shot baseline. Key metrics showed significant relative improvements: Word Error Rate (WER) improved by 78.26%, Character Error Rate (CER) by 93.10%, and Match Error Rate (MER) by 57.0%. Beyond raw transcription accuracy, the model showed improved semantic coherence, reflected in higher BLEU and METEOR scores. Furthermore, the researchers observed a 96.70% reduction in the hallucination rate, suggesting that controlled fine-tuning effectively mitigates the tendency of the base model to generate non-existent or incorrect text when processing Assamese speech.

Assamese is a morphologically rich language with unique phonological features and a complex classifier system, yet it remains underrepresented in global ASR research. By establishing a robust, hardware-efficient fine-tuning framework, this study provides a practical pathway for developing speech-based interfaces in languages with limited digital resources. The findings demonstrate that even with modest amounts of annotated data, strategic fine-tuning of large transformer-based models can bridge the performance gap between high-resource and low-resource languages, facilitating better accessibility for education and government services.