SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning Paradigms for LLMs | Kejian Zhu et al. | ResearchPod

Transcript: SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning Paradigms for LLMs

Alex: Welcome to another episode of ResearchPod. Today, we're discussing a study that tackles a frustrating problem in AI development: why does teaching a model a new skill so often cause it to forget an old one?

Sam: So this paper is basically asking why training an AI on coding might make it worse at math—even though you never touched the math training?

Alex: Exactly. And the researchers found that the answer depends almost entirely on how you train the model, not just what you train it on. There are two common methods, and they behave very differently when a model needs to hold onto multiple skills at once.

Sam: What are those two methods?

Alex: The first is called Supervised Fine-Tuning. Think of it like a teacher handing a student a completed answer sheet and saying, "Copy this exactly." The model sees thousands of correct examples and adjusts itself to match them as closely as possible.

Sam: And the problem is that adjusting to match the new examples messes up the adjustments you already made for the old ones?

Alex: That's precisely it. The researchers call this "SFT Conflict." When you fine-tune on coding after fine-tuning on math, the model reshapes itself to fit the coding examples—and in doing so, it shifts away from what it learned about math. It's a bit like repainting a wall. The new color covers the old one completely.

Sam: So what does the second method do differently?

Alex: The second method is called Reinforcement Learning. Instead of copying a correct answer, the model tries to solve a problem on its own and then receives a signal telling it whether it did well or poorly. It's more like trial and error—the model experiments, gets feedback, and gradually figures out what works.

Sam: Like a video game where you learn the level by playing it, rather than reading a walkthrough.

Alex: That's a good comparison. And here's why it matters for the forgetting problem. Because the model is only rewarded for what actually helps it score better, it ends up making very small, very targeted adjustments. It doesn't rewrite everything—it just fine-tunes the specific parts that are relevant to the task at hand.

Sam: The study used the word "orthogonal" to describe these updates. That sounds technical. What does it actually mean here?

Alex: Right, so before we use that word—imagine a bookshelf packed with books. If you try to shove a new book in, you have to push the others aside. The whole shelf gets disrupted. That's what happens with the first method: new training displaces old knowledge.

SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning Paradigms for LLMs

Kejian Zhu, Zhuoran Jin, Shangqing Tu, Hongbang Yuan, Yushi Bai, Kang Liu, Juanzi Li, Jun Zhao

Aug 4, 2026·4:20·arxiv·Original Paper

SFT ConflictsRL CoexistsMulti-task Gradient InterferenceParallel-RLMulti-Stage SFT Performance CollapseGradient Orthogonality in RL

Listen

4 min

Briefing

Abstract

Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) exhibit fundamentally different behaviors in enhancing multi-task reasoning for large language models (LLMs). Our preliminary experiments revealed a phenomenon: SFT suffers from severe task conflicts under multi-stage training, whereas RL enables stable coexistence across diverse tasks. Empirically, we trace this to the parameter level, observing that RL induces sparse and approximately orthogonal updates across tasks. We provide a theoretical explanation for this mechanism by analyzing multi-task gradient interference. Our results reveal a distinction: interference in SFT is norm-limited, scaling with the absolute gradient magnitude, whereas interference in RL is variance-limited, bounded by the gradient variance induced by advantage normalization and on-policy optimization. This small variance bound yields near-orthogonal optimization directions across tasks. Leveraging this insight, we propose Parallel-RL, a paradigm that decouples multi-task training, significantly improving efficiency and flexibility.

Research Question and Approach

This paper investigates why Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) behave differently when training Large Language Models (LLMs) on multiple tasks. While SFT is typically performed on mixed datasets to avoid performance degradation, RL is often applied in multi-stage settings where tasks are learned sequentially. The authors conduct an empirical and theoretical analysis to determine why SFT experiences catastrophic forgetting or performance collapse in multi-stage training, while RL maintains or improves performance across tasks.

Empirical Findings

The authors observe that SFT and RL exhibit fundamentally different parameter update dynamics. In SFT, updates are dense and have large magnitudes, leading to high cosine similarity between task-specific updates—essentially, the model 'unlearns' previous tasks to accommodate new ones. In contrast, RL updates are sparse, minimal in magnitude, and exhibit near-zero cosine similarity across different tasks. This orthogonality means that RL updates for one task occupy a different subspace than those for another, allowing the model to accumulate capabilities without interfering with previously learned knowledge.

Theoretical Mechanism

The authors provide a theoretical framework to explain these observations. They identify two key drivers for RL's stability: the advantage function and the on-policy nature of the training. The advantage normalization in RL (specifically in GRPO) enforces a zero-sum property that removes the mean gradient direction, effectively filtering out the large, overlapping components found in SFT. Furthermore, because RL is on-policy, the updates are constrained by the current model's distribution, leading to a KL-minimal solution that keeps parameter changes small and sparse. SFT, being off-policy and driven by an external expert distribution, lacks these constraints, resulting in larger, more disruptive gradient interference.

Parallel-RL

Leveraging the insight that RL updates are approximately orthogonal, the authors propose 'Parallel-RL.' This paradigm decouples multi-task training by training tasks independently in parallel and merging their updates. This approach improves training efficiency and flexibility while maintaining the performance gains of single-task RL, confirming that the orthogonality of RL updates is a robust mechanism for multi-task learning.

Key Terms and Definitions

SFT Conflicts — A phenomenon where optimizing a model for a new task using supervised learning causes significant performance degradation on previously learned tasks.
RL Coexists — A phenomenon where RL-based training allows a model to acquire new task capabilities while preserving or enhancing performance on existing tasks.
Gradient Interference — The degree to which the parameter updates for one task negatively impact the performance of another task, measured by the inner product of their gradient vectors.
On-Policy — A training paradigm where the model learns from data generated by its current policy, which the authors show leads to smaller, more stable updates compared to off-policy SFT.
Advantage Normalization — A technique in RL that centers rewards within a group of rollouts, which the authors identify as a primary mechanism for reducing gradient interference between tasks.

SFT Conflicts, RL Coexists: A Theoretical and Empirical Analysis of Multi-Task Learning Paradigms for LLMs: Research Paper Podcast