ChatShopBuddy: Towards Reliable Conversational Shopping Agents via Reinforcement Learning | Yiruo Cheng et al. | ResearchPod

Transcript: ChatShopBuddy: Towards Reliable Conversational Shopping Agents via Reinforcement Learning

Alex: Welcome to another episode of ResearchPod.

Sam: Today we're looking at a paper called "ChatShopBuddy: Towards Reliable Conversational Shopping Agents via Reinforcement Learning." It explores using reinforcement learning—a training method that rewards good actions—to build better AI shopping helpers. These agents handle conversations, not just searches, while juggling goals like picking right products and explaining them without inventing facts.

Alex: So these are AI chat systems for shopping, like suggesting camping gear for a cozy family trip? The issue is they sometimes give wrong info or skip details, even if they sound convincing?

Sam: Yes. Picture asking: "It's our first family camping trip—suggest things for a cozy, fun mood." A typical AI might recommend lights and chairs smoothly, but pick wrong items or make up prices. The paper finds these fluent errors, called hallucinations, common in shopping chats. They erode trust, and generic large language models—which are huge AI systems trained on vast text—struggle with limits like budgets or specific needs.

Alex: Users want solid advice, not just smooth talk. Why are shopping chats so tricky?

Sam: Shopping goes beyond simple lookups. Users ask vague or multi-part questions, like item bundles or comparisons with constraints. Large language models often focus on sounding human over staying accurate, so results vary wildly.

Alex: The core challenge is balancing facts with smooth, efficient responses?

Sam: Right. The researchers train agents systematically, prioritizing basics like product accuracy before extras like clear explanations or quick tool use. This yields steadier performance than untrained larger models.

Alex: They enforce facts first. How do they structure that training?

Sam: They layer rewards like a school report card: pass basic checks before scoring style. Only correct products and real details earn points for explanations or speed. Researchers call this Hierarchical Reward Modeling, or HRM. It prevents the AI from faking fluency on wrong facts.

Alex: Like a solid foundation before walls. What do those basic checks cover?

Sam: The first layer verifies products match needs, text stays on topic, and descriptions use real data—no inventions. A second layer scores organization and detail, but only if basics pass. These are L1 and L2 checks for reliability.

Alex: For a quiet blender under $100 that crushes ice, a regular AI might ignore limits and hype a bad fit?

Sam: Exactly. Generic models skip constraints and invent specs to persuade. HRM blocks fancy rewards until real matches appear, creating steadier advice. The paper notes a smaller trained model outperforms bigger untrained ones in consistency—not just higher peaks, but fewer lows.

ChatShopBuddy: Towards Reliable Conversational Shopping Agents via Reinforcement Learning

Yiruo Cheng, Kelong Mao, Tianhao Li, Jiejun Tan, Ji-Rong Wen, Zhicheng Dou

Mar 6, 2026·6:29·arxiv·Original Paper

Conversational Shopping AgentsReinforcement Learning (RL)SmartShopBenchHierarchical Reward Modeling (HRM)Dynamic Contrastive Policy Optimization (DCPO)Conversational Shopping Agent

Listen

6 min

Briefing

Abstract

Conversational shopping agents represent a critical consumer-facing application of Large Language Model (LLM)-powered agents, yet how to effectively apply post-training Reinforcement Learning (RL) to optimize such agents remains underexplored. This work investigates RL-based optimization for shopping agents in real-world scenarios, where agents must simultaneously satisfy multiple interdependent objectives spanning objective metrics (product correctness), subjective qualities (persuasiveness), outcome rewards (final response quality), and process rewards (tool efficiency). We present a complete methodology to address this challenge. Specifically, we first construct SmartShopBench, a benchmark that captures diverse shopping intents with a hierarchical evaluation that decomposes complex quality requirements into measurable levels. Building on this evaluation framework, we design Hierarchical Reward Modeling (HRM) to structure mixed reward types through conditional gating that reflects their logical dependencies. To enable efficient training, we further propose Dynamic Contrastive Policy Optimization (DCPO), which balances response quality with operational efficiency through dynamic trajectory selection based on reward and reasoning length. Extensive experiments demonstrate that our RL-trained agent, namely ChatShopBuddy, consistently outperforms larger models relying on generic reasoning, achieving superior stability rather than merely higher peaks. Our work provides valuable guidance for applying RL to real-world conversational agents.

ChatShopBuddy: Towards Reliable Conversational Shopping Agents via Reinforcement Learning: Research Paper Podcast

ChatShopBuddy: Towards Reliable Conversational Shopping Agents via Reinforcement Learning

Listen