Scaling Laws for Task-Specific LLM Distillation

Research Question and Approach

This paper investigates the trade-offs involved in compressing large language models (LLMs) for domain-specific applications. Specifically, the authors seek to quantify how different compression strategies—such as iterative structural pruning and knowledge distillation—affect the balance between maintaining high performance on a narrow, domain-specific task (financial news event classification) and preserving the model's broader general-knowledge capabilities. The researchers derive empirical scaling laws by varying dataset size, compression ratio, supervision format, and pruning schedules, using a 32B-parameter model as the teacher.

Methodology

To address the challenges of model compression, the authors employ a combination of lightweight Neural Architecture Search (NAS) for joint depth and width pruning and various distillation techniques. They compare LoRA-based and logit-based distillation, introducing a novel "blended chain-of-thought (CoT) supervision loss." This loss function allows for explicit reweighting between label tokens and reasoning traces during KL-divergence distillation, which the authors found necessary to stabilize training when using CoT. The experiments are grounded in a synthetic financial dataset, FinHeadlineMix, designed to reflect industry-standard production workloads.

Key Findings

The study reveals that supervision format is the primary driver of the performance trade-off during compression. While in-domain task quality degrades predictably as the model is pruned, general-knowledge benchmarks collapse significantly earlier. Crucially, the authors demonstrate that using blended CoT supervision allows the student model to actively recover general knowledge that is otherwise lost during the pruning process. Furthermore, the researchers find that the training process itself accounts for a substantial portion of the performance gap, and that iterative pruning can successfully compress the teacher model to 16% of its original parameter count while retaining meaningful task accuracy.

As LLMs become increasingly integrated into latency- and cost-sensitive production environments, practitioners face the difficult challenge of shrinking models without sacrificing their utility. This paper provides a reusable framework and practical recommendations for domain-specific compression, helping researchers and engineers make informed decisions about how to allocate compute and choose distillation methods to balance task-specific performance with general model robustness.