Dongkyu Cho, Rumi Chunara, Ali Behrouz
8 min
Abstract
Continual learning (CL) seeks to adapt pretrained models while preserving prior capabilities. Yet the tasks supplied for preservation cover only part of what a model knows, while forgetting can extend beyond them. Recent work shows that language models can supplement this limited evidence by generating useful replay without access to their original training data. This raises a question about self-preservation: can a model help choose preservation targets that protect its own capabilities more broadly? To study this question, we organize task descriptions and their examples into Task Banks, whose entries can be selected and combined to guide replay generation by a frozen model. Continual instruction tuning with Qwen models of varying sizes yields three findings. First, the new task and learning rate shape which prior capabilities degrade, while tracking previously correct answers reveals losses that average accuracy can conceal. Second, generated replay reduces forgetting on tasks outside the bank. Combining partial banks strengthens this protection, improving mean retention of previously correct answers over either bank alone across the tested sizes. Third, the model can also help choose replay targets: selection informed by the new task improves mean held-out retention over random selection at 4B, with similar new-task gains, although smaller models show no corresponding advantage. Under a fixed example budget, however, replacing new-task examples with replay improves preservation at the cost of smaller new-task gains. Together, these findings show that models can support broader self-preservation by generating replay data and, in some settings, choosing which tasks to replay.
Sam: Before results, how are they scoring this? Average accuracy can look fine while a lot happens underneath.
Alex: That's their first finding. In one Qwen3-4B MedQA run, about three and a half percent of held-out answers flipped from right to wrong. A nearly identical share flipped from wrong to right. Net change was basically zero. So they report Retention: the share of answers that were correct before training and are still correct after.
Sam: So a flat average can hide real churn in what the model knows. Which is the thing you'd actually care about.
Alex: And they found the incoming task and learning rate shape which skills degrade. In one sweep, a lower learning rate gave both a larger new-task gain and much better retention than a higher one. Choosing a gentler rate isn't automatically a trade-off.
Sam: Okay, now the replay itself. Does model-generated data actually protect the held-out tasks?
Alex: In one comparison, Qwen3-4B learns a stance-detection task in Chinese, called C-STANCE. Plain fine-tuning lost about four points on average across held-out tasks. Replaying a single old task barely helped. Replaying all four bank tasks turned that into a small gain, about one point, whether the replay was real stored data or model-generated. And new-task learning stayed essentially the same.
Sam: That's the broad-versus-narrow point. What happens when you combine partial banks?
Alex: They split the bank in half. One half is knowledge and math, the other is commonsense and reading. Then they compared each half against the two combined, on MedQA, at all three model sizes. The combined bank had the best mean held-out Retention every time, roughly three to four points above the better half. At 4B it kept about 96 percent of previously correct answers. New-task gains dropped by less than a point relative to plain fine-tuning.
Sam: You said mean. Did combining win on every held-out task?
Alex: No. The best bank varied by task. The combined bank wins on the average, not uniformly.
Sam: Now the part about the model choosing. If it picks two of the four tasks, does it beat random?
Alex: At 4B, yes. When the model saw a description and examples of the incoming task, its chosen pair kept about 96 percent of correct held-out answers. A random pair kept about 93. Seeing the bank alone, without the incoming task, did little better than random. And the informed pair came within a fraction of a point of using all four.
Sam: That's one model size. What about the smaller ones?
Alex: No advantage. At 0.6B the informed choice was slightly worse than random. At 1.7B, roughly a tie. The authors are explicit that the benefit is limited to the tested 4B setting. They also didn't vary the strength of the learner, the selector and the generator separately, so we can't say which one drives it.
Sam: Here's my practical worry. Replay costs something. If I have a fixed training budget, what do I give up?
Alex: They tested that directly. With the number of examples held fixed, replay replaced some new-task examples. Generated replay cut the worst held-out loss by more than half. But the MedQA gain also fell by more than half. The authors are careful here. The benefit can't be credited to replay content alone, because replay also reduced exposure to the new task. Token counts and generation cost weren't matched either.
Sam: So under a tight budget, it's a trade, not a free lunch.
Alex: That's how they frame it. They don't claim a better trade-off curve.
Sam: What about letting the model propose new tasks for the bank? That seems like the natural next step.
Alex: They tried it as a single pilot, and it went badly. Accuracy dropped on the new task and all four held-out tasks. Part of that was output formatting, the math answers stopped parsing. But it's one run at one setting, so it says nothing either way about discovery's value. A two-step sequential pilot was also mixed.
Sam: Then how far should I trust the main results overall?
Alex: I'd call it moderate. The direction is consistent across sizes. But it's one model family, two incoming tasks, eight benchmarks, mostly a single update. Main studies used five seeds, some only two. Generated answers aren't verified, so replay can reproduce the model's own errors. And held-out transfer doesn't show the full pretrained repertoire survived.
Sam: So who should actually read the full paper, and where should they start?
Alex: People working on continual learning or post-training. Start with the bank-composition and target-selection results, then the fixed-allocation trade-off. Read the limitations appendix carefully. It's candid about what wasn't controlled.
Sam: And for everyone else, the one thing to carry around?
Alex: Count the right answers you lose, not just the average. And when you rehearse, rehearse broadly, because protection spills beyond what you targeted.
Sam: Worth keeping in mind before the next fine-tune.