Can Scale Save Us From Plasticity Loss in Large Language Models?

This paper investigates whether modern GPT-style Transformer models suffer from loss of plasticity—the gradual decline in a neural network's ability to learn new information after extended training. While this phenomenon is well-documented in smaller, older architectures, it remains unclear if it persists in large-scale language models trained on realistic natural-language data.

The authors designed a multilingual continual learning framework using eight languages from the CulturaX dataset. They trained GPT-style, pre-norm decoder-only Transformers ranging from 5M to 314M non-embedding parameters. To measure plasticity, they periodically paused the continual pretraining to evaluate the model's ability to learn a held-out Vietnamese probing task. By comparing the area under the validation-loss curve (AUC) during these probing sessions, the researchers could quantify how effectively each model adapted to new data over time.

The experiments demonstrate that plasticity loss is a persistent issue in Transformer models, regardless of scale. While larger models exhibit more initial beneficial transfer—meaning they learn the probing task more efficiently than smaller models—they still eventually experience a decline in learning performance. The onset of this plasticity loss follows a predictable, sublinear power-law scaling with model size, suggesting that simply increasing parameter count is an insufficient strategy to permanently maintain plasticity. Furthermore, the researchers observed this degradation even under stationary training conditions, indicating that plasticity loss is not solely a byproduct of abrupt task switching.