The Disconnect Between Better Descriptive Reasoning Trace Quality and Recommendation Effectiveness

This paper investigates whether improving the quality of explicit natural-language descriptive reasoning traces in generative recommendation translates into better traditional offline recommendation effectiveness. The authors conduct the first controlled 2x2 factorial study comparing item representation (Semantic IDs vs. Natural-Language Titles) and reasoning integration (No Reasoning vs. Reasoning) using a shared Qwen3-1.7B backbone across three Amazon product domains.

The study evaluates models under a rigorous three-stage pipeline: Stage 1 trains lightweight no-reasoning baselines using LoRA; Stage 2 warm-starts from these baselines and performs full supervised fine-tuning (SFT) using a 50/50 mix of reasoning and direct prediction examples with GPT-generated traces; and Stage 3 applies Group Relative Policy Optimization (GRPO) reinforcement learning with a prefix-match reward. This design isolates the effects of item representation and semantic grounding while keeping model architecture, supervision, and teacher traces constant.

The central finding reveals a disconnect: while natural-language titles and extensive semantic ID alignment both produce substantial improvements in descriptive reasoning trace quality, neither consistently improves traditional offline recommendation effectiveness. Reasoning over semantic IDs yields generic and poorly grounded traces, whereas reasoning over titles produces highly interpretable and grounded traces, yet neither configuration yields a positive reasoning delta under standard training objectives. However, optimizing with a richer reward signal partially recovers recommendation performance, demonstrating that optimization plays a crucial role.