ResearchPod Summary
This paper investigates why Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) behave differently when training Large Language Models (LLMs) on multiple tasks. While SFT is typically performed on mixed datasets to avoid performance degradation, RL is often applied in multi-stage settings where tasks are learned sequentially. The authors conduct an empirical and theoretical analysis to determine why SFT experiences catastrophic forgetting or performance collapse in multi-stage training, while RL maintains or improves performance across tasks.
The authors observe that SFT and RL exhibit fundamentally different parameter update dynamics. In SFT, updates are dense and have large magnitudes, leading to high cosine similarity between task-specific updates—essentially, the model 'unlearns' previous tasks to accommodate new ones. In contrast, RL updates are sparse, minimal in magnitude, and exhibit near-zero cosine similarity across different tasks. This orthogonality means that RL updates for one task occupy a different subspace than those for another, allowing the model to accumulate capabilities without interfering with previously learned knowledge.
The authors provide a theoretical framework to explain these observations. They identify two key drivers for RL's stability: the advantage function and the on-policy nature of the training. The advantage normalization in RL (specifically in GRPO) enforces a zero-sum property that removes the mean gradient direction, effectively filtering out the large, overlapping components found in SFT. Furthermore, because RL is on-policy, the updates are constrained by the current model's distribution, leading to a KL-minimal solution that keeps parameter changes small and sparse. SFT, being off-policy and driven by an external expert distribution, lacks these constraints, resulting in larger, more disruptive gradient interference.
Leveraging the insight that RL updates are approximately orthogonal, the authors propose 'Parallel-RL.' This paradigm decouples multi-task training by training tasks independently in parallel and merging their updates. This approach improves training efficiency and flexibility while maintaining the performance gains of single-task RL, confirming that the orthogonality of RL updates is a robust mechanism for multi-task learning.
Alex: Welcome to another episode of ResearchPod. Today, we're discussing a study that tackles a frustrating problem in AI development: why does teaching a model a new skill so often cause it to forget an old one?
Sam: So this paper is basically asking why training an AI on coding might make it worse at math—even though you never touched the math training?
Alex: Exactly. And the researchers found that the answer depends almost entirely on how you train the model, not just what you train it on. There are two common methods, and they behave very differently when a model needs to hold onto multiple skills at once.
Sam: What are those two methods?
Alex: The first is called Supervised Fine-Tuning. Think of it like a teacher handing a student a completed answer sheet and saying, "Copy this exactly." The model sees thousands of correct examples and adjusts itself to match them as closely as possible.
Sam: And the problem is that adjusting to match the new examples messes up the adjustments you already made for the old ones?
Alex: That's precisely it. The researchers call this "SFT Conflict." When you fine-tune on coding after fine-tuning on math, the model reshapes itself to fit the coding examples—and in doing so, it shifts away from what it learned about math. It's a bit like repainting a wall. The new color covers the old one completely.
Sam: So what does the second method do differently?
Alex: The second method is called Reinforcement Learning. Instead of copying a correct answer, the model tries to solve a problem on its own and then receives a signal telling it whether it did well or poorly. It's more like trial and error—the model experiments, gets feedback, and gradually figures out what works.
Sam: Like a video game where you learn the level by playing it, rather than reading a walkthrough.
Alex: That's a good comparison. And here's why it matters for the forgetting problem. Because the model is only rewarded for what actually helps it score better, it ends up making very small, very targeted adjustments. It doesn't rewrite everything—it just fine-tunes the specific parts that are relevant to the task at hand.
AI-generated third-party summary by ResearchPod. Not official content or an endorsement by the paper authors or affiliated organizations.
Sam: The study used the word "orthogonal" to describe these updates. That sounds technical. What does it actually mean here?
Alex: Right, so before we use that word—imagine a bookshelf packed with books. If you try to shove a new book in, you have to push the others aside. The whole shelf gets disrupted. That's what happens with the first method: new training displaces old knowledge.
Sam: And the second method?
Alex: With Reinforcement Learning, it's as if instead of shoving a new book in, you install a small, dedicated hook on a completely empty patch of wall. The new item has its own space. It doesn't touch anything else. When the researchers say the updates are "orthogonal," they mean the changes made for one task are essentially invisible to the other tasks—they occupy different territory inside the model, so they can't interfere.
Sam: So the model isn't forgetting because it's running out of space—it's forgetting because the first method keeps writing over the same space, while the second method finds fresh space each time.
Alex: That's the core insight. And it has a practical consequence the researchers took seriously. If Reinforcement Learning updates genuinely don't interfere with each other, you could train separate models on separate tasks and then mathematically combine them—because the underlying guarantee means they won't conflict when merged.
Sam: Is that the "Parallel-RL" idea they proposed?
Alex: It is. Rather than training one model on task after task in sequence—where each new task risks overwriting the last—you train multiple models in parallel, each on a different task, and then merge the results. The math the researchers provide suggests this should be stable, because the updates from each task are pointing in different directions and don't cancel each other out.
Sam: That's a meaningful shift in how you'd think about building these systems. Instead of one model that learns everything in order, you'd have specialists that get combined.
Alex: And the broader point the paper makes is that the mechanism of learning matters as much as the data itself. Two models could be trained on identical information, but if one copies answers and the other learns through feedback, they'll end up with very different internal structures—and very different abilities to hold onto multiple skills. Understanding that distinction is what opens the door to more reliable, multi-task AI systems.
Sam: It makes you realize how much is happening beneath the surface when these models are trained—choices that seem technical end up having real consequences for what the model can and can't do.
Alex: That's a good way to leave it. Thanks for listening to ResearchPod.