Jack Lu, Zhenbang Yang, Mike Lasby, Tejas Pote, Yani Ioannou, Mengye Ren
8 min
Abstract
How little of a pretrained large language model (LLM) has to change for it to acquire new capabilities? We find that zeroing fewer than 0.05% of its weights, with no other modification, matches full fine-tuning (FFT) at scale across three LLM post-training regimes: reinforcement learning with verifiable rewards, supervised fine-tuning, and on-policy distillation. Our method, Deletion Tuning, learns a binary mask over the gate projection of transformer blocks. Two practical advantages follow: the trained mask is hundreds to thousands of times smaller than the corresponding FFT checkpoint, and Deletion Tuning forgets far less, which addresses a central challenge in continual learning. Under reinforcement learning, its prior-task accuracy stays near the base level while FFT, LoRA, and random sparse fine-tuning lose up to 5.4 points. To understand why masking alone suffices, we analyze the geometry of the learned weight changes under reinforcement learning using stable rank, spectrum drift, and principal-weight overlap. Compared with LoRA and random sparse fine-tuning, Deletion Tuning is closest to FFT across the three measures.
Alex: That distinction matters. During training, weights can be partially suppressed. But the reported deployed model has only the original values and zeros?
Sam: Yes. They evaluate using the hard mask—the collection of keep-or-erase decisions. Their check finds almost no accuracy difference between the final smooth version and that hard version. So the reported result isn’t relying on hidden fractional adjustments at deployment.
Alex: Where do they put these masks? Choosing the location seems almost as important as choosing what to erase.
Sam: They target the gate projection, a weight matrix controlling intermediate signals inside each transformer block. In these models, that pathway multiplicatively controls signals passing through the block’s feed-forward component. Other model weights stay frozen, and training chooses the deletion rate rather than enforcing a fixed quota.
Alex: Give us the concrete result. How little disappears, and what does the adapted model manage to do?
Sam: In every main reinforcement-learning run, fewer than zero point zero five percent of all weights are deleted. On the larger Llama model’s school-math benchmark, deletion and full fine-tuning both solve a little over four out of five problems.
Alex: If the strongest result came from one model and one math dataset, I’d hesitate to generalize. How broad is the test?
Sam: They test Llama and Qwen models, including held-out math benchmarks. At larger scales, deletion matches full fine-tuning, including a mixture-of-experts model with thirty billion total parameters. The smaller models generally show a gap, so this is not uniform equivalence at every size.
Alex: And reinforcement learning might be unusually suited to small edits. Does the idea survive training that supplies more detailed targets?
Sam: They also test supervised fine-tuning, where the model imitates fixed target responses. And they test on-policy distillation, where a student samples responses and learns to match a teacher’s token predictions. At the largest tested scale within each setting, deletion matches full fine-tuning on the reported tasks.
Alex: That supports a broader method, rather than only a math-training trick. But could the initial mask already account for most of the improvement?
Sam: They check that. Initialization uses Wanda, a pruning method that scores weights using their magnitudes and input activations. Wanda alone stays close to the base model, even when the authors select its best tested sparsity. Learning the mask produces much larger gains.
Alex: What about the phrase “new capabilities”? Better accuracy could mean making existing solutions more likely, without solving anything genuinely different.
Sam: They sample many answers per problem to examine that distinction. Even with one hundred twenty-eight attempts, the base models miss problems that the deletion-tuned models solve. That supports a gain beyond simply improving the chance of selecting an already frequently produced answer, although finite sampling cannot prove the base model could never solve them.
Alex: Let’s separate performance from deployment. What does the small mask mean for someone keeping different versions for different tasks?
Sam: In the larger Llama math comparison, the compressed mask occupies about eight megabytes. The full fine-tuned checkpoint occupies about thirty gigabytes. The mask is also smaller than the smallest LoRA adapter tested in that comparison.
Alex: That’s a storage result. Does it also mean cheaper training or faster generation?
Sam: Not automatically. Training keeps a score for every targeted weight, so memory use is lower than full fine-tuning but higher than LoRA. And these scattered deletions are not a demonstrated inference-speed improvement. The authors propose more structured masks as future work.
Alex: The other practical claim was less forgetting. How did they test whether old skills survived learning the new task?
Sam: During math reinforcement learning, they track performance on a separate suite covering knowledge, reasoning, instruction following, and coding. Deletion stays near the base model across the tested Llama sizes. Competing methods lose as much as about five percentage points.
Alex: Is that just because deletion changes fewer weights? The random sparse method seems like a useful check on that explanation.
Sam: One sparse baseline changes a comparable fraction of weights and still forgets more. That suggests edit count alone doesn’t explain retention. The authors also measure how much output probabilities drift from the base model, and deletion shows much less drift.
Alex: Do they explain why such restricted edits can compete with full fine-tuning, or is the mechanism still mostly open?
Sam: They compare the geometry of weight changes: how many directions an update uses, how much the matrix’s singular values shift, and where changes fall. Across those measures together, deletion resembles full fine-tuning more closely than the tested alternatives. That helps explain the result, but it isn’t a proof of why deletion succeeds.
Alex: So how far should we trust the practical promise, and who should read the full document?
Sam: I find the cross-model and cross-objective evidence persuasive as a feasibility demonstration. But these are best-checkpoint evaluations in a preprint, and sequential continual learning remains untested. Preserving skills while learning one task does not establish that repeated deletion will keep working across a long task sequence.
Alex: If someone is choosing an adaptation method now, where should they start reading?
Sam: Post-training researchers should start with the Deletion Tuning method and the reinforcement-learning comparisons, including the smaller-model gaps. Continual-learning researchers should read the forgetting section alongside the limitations. Everyone else can keep this distinction: in these tests, learning sometimes meant choosing what to erase, not writing something new.
Alex: A useful possibility, with the repeated-learning test still ahead.