Towards Fast Domain Adaptation and Fine-Grained User Simulation for Evaluating Conversational Recommender Systems

Conversational Recommender Systems (CRSs) are increasingly used to provide personalized recommendations through natural language. However, evaluating these systems is difficult because they are dynamic and interactive. While researchers have turned to Large Language Model (LLM) based user simulators to automate this process, current simulators are limited by rigid, manually crafted prompts, predefined action spaces, and a lack of fine-grained control over linguistic style. Furthermore, existing evaluation methods often suffer from biases, such as flattery or rough scoring, and fail to capture turn-level errors or system robustness.

AdaptSim is designed to overcome these limitations through four primary mechanisms:

1. Automatic Prompt Optimization: Instead of manual engineering, AdaptSim uses domain-specific knowledge and interaction data to iteratively refine prompts, enabling faster adaptation to new domains.
2. Open-ended Action Generation: By moving away from fixed action spaces, the simulator dynamically constructs user actions based on the current dialogue context, increasing flexibility.
3. Thinking-Enhanced Style Generation: The simulator employs a "think-then-respond" strategy, where the model explicitly plans its linguistic style (e.g., tone, sentiment, politeness) before generating a response, ensuring stylistic consistency.
4. BFS-based Turn-level Pairwise Evaluation: To address the flaws in pointwise scoring, AdaptSim uses a Breadth-First Search (BFS) approach to compare dialogue trajectories. This allows for an objective, turn-level assessment that captures both basic recommendation capabilities and system robustness against noisy or varied user inputs.

AdaptSim provides a more scalable and reliable way to evaluate CRSs. By reducing the need for manual prompt design and enabling the simulation of diverse, realistic user behaviors, it allows developers to test how robust their systems are under different conditions. The turn-level pairwise evaluation framework also offers a more diagnostic approach to identifying exactly where a CRS fails during a conversation, rather than relying on coarse, dialogue-level metrics that often mask performance issues.