SmartGR: Hierarchy and Beam-Aware Knowledge Distillation for Generative Recommendation

Generative recommendation (GR) models have demonstrated significant performance gains by scaling up, but this comes at the cost of high inference latency. While knowledge distillation (KD) is a standard approach for model compression, existing methods fail to address the unique structural challenges of GR: the hierarchical nature of semantic IDs (SIDs) and the specific mechanics of beam search decoding. This paper asks how to effectively distill a large teacher GR model into a lightweight student while accounting for these two factors.

The authors propose SmartGR, a distillation framework that introduces two novel objectives to align the student model with the teacher's capabilities:

1. Hierarchy-Aware SID Distillation: Recognizing that distillation difficulty varies across the coarse-to-fine SID hierarchy, this method assigns learnable, depth-dependent weights to the distillation loss at each SID position. This ensures the student prioritizes learning from the teacher where the teacher's advantage is most pronounced.

2. Beam-Aware Ranking Distillation: To prevent the student from pruning high-quality items during beam search due to low intermediate prefix scores, this objective forces the student to mimic the teacher's relative ranking preferences between positive and negative beam paths. By distilling the teacher's cumulative score distributions, the student learns to better preserve promising candidates throughout the generation process.

Experiments across four benchmark datasets (Amazon Beauty, Amazon Toys, Kuaishou Ad, and Kuaishou Video) show that SmartGR consistently outperforms existing distillation baselines. By specifically targeting the hierarchical SID structure and the beam search mechanism, SmartGR achieves an average performance improvement of 8.6% over standard distillation methods while simultaneously providing a 2.39x speedup in inference time. The results demonstrate that accounting for the specific generation pipeline of GR models is critical for effective model compression.