Unified Gradient Projection: Language-Balanced Continual Learning for Multilingual Low-Resource ASR

As large-scale foundation models like Whisper are adapted to new, low-resource languages, they face the classic plasticity-stability dilemma: learning new languages often overwrites representations critical for previously learned ones, a phenomenon known as catastrophic forgetting. The authors investigate how to regulate cross-task interference in these multilingual settings, where dominant languages often bias the optimization process and exacerbate forgetting in under-represented languages.

The authors propose Unified Gradient Projection (UGP), a continual learning framework that combines two mechanisms:

1. Language-Balanced Gradient Regulation: Instead of using raw gradients, UGP computes a reference gradient from a balanced mixture of historical language samples. This reference gradient is used to project the current training gradient into an orthogonal subspace, ensuring that updates do not conflict with previously learned knowledge.
2. Experience Replay (ER) Integration: UGP incorporates standard experience replay to provide data-level stability, which acts as a form of joint multilingual training that encourages positive knowledge transfer across languages.

By combining these, UGP ensures that the optimization trajectory is both constrained by past knowledge (via projection) and supported by historical data (via replay).

UGP demonstrates superior performance across various model scales, from Whisper-small to Whisper-large-v3. On the large-v3 model, UGP achieves near-zero average forgetting (0.04% FWER) while maintaining high plasticity for new target languages. The authors show that UGP effectively reshapes the optimization landscape, leading to gradient interactions that are closer to orthogonal, which prevents the destructive interference common in standard fine-tuning. Furthermore, UGP remains robust even under extreme data scarcity, such as when only 5 hours of training data are available per language.