Designing Reward Signals for Portable Query Generation: A Case Study in Industrial Semantic Job Search

Modern job-search platforms often struggle with the lexical gap between a candidate's detailed profile and the concise, high-level queries required for effective search. The authors address this by developing an RLAIF (Reinforcement Learning from AI Feedback) framework to generate 'portable' queries—search terms that abstract away specific identifiers (like employer names or locations) while retaining the candidate's core qualifications. The central challenge is preventing the model from 'reward hacking,' where it learns to simply copy text from the profile to satisfy the LLM-as-judge rubric.

To isolate the impact of different variables, the researchers compared four on-policy optimization algorithms: PPO (with a critic), GRPO, RLOO, and REINFORCE++. They implemented a two-part reward signal: a rubric-based LLM judge and a deterministic, rule-based 'reward floor' designed to penalize verbatim copying. The study utilized a three-layer evaluation framework, culminating in an independent, cross-family judge (Llama-3.3-70B-Instruct) to mitigate the bias inherent in using the same model for training and evaluation.

• Algorithm Parity: Once the reward signal is properly shaped, the performance delta between critic-free optimizers (GRPO, RLOO, REINFORCE++) is statistically negligible, rendering the specific choice of algorithm largely immaterial.
• Reward Shaping: The deterministic reward floor was the most significant driver of quality, providing a +0.147 improvement on the evaluation judge. Without this, GRPO was particularly susceptible to exploiting spurious reward signals.
• Evaluation Inflation: The training-time judge consistently overestimated performance gains by 2.4x compared to the independent evaluation judge, highlighting the danger of relying solely on internal training metrics.