FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning | Yijun Pan et al. | ResearchPod

Transcript: FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning

Alex: Welcome to another episode of ResearchPod.

Sam: Today we're looking at a paper called FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning. The central puzzle is training large language models—AI systems like chatbots that handle human-like text—to make product or content suggestions that shift with user needs, using reinforcement learning.

Alex: So it tackles how typical suggestion systems, built for one goal like more clicks, struggle when user wants change quickly?

Sam: Yes. Traditional systems suggest movies or videos based on past likes for a fixed aim, like maximizing clicks. They can't switch easily without retraining. But users shift intents fast—one day they want discoveries, the next favorites—while businesses might push trends mid-day.

Alex: And large language models adapt by just telling them new goals in words?

Sam: Right—their general training gives strong reasoning, so prompts help without full retraining. But for recommendations, the paper uses reinforcement learning from verifiable rewards. The AI tries suggestions, gets scores based on real matches to needs—like how well the list fits—and learns from that trial and error.

Alex: But users only click a few items, right? That leaves sparse, noisy feedback.

Sam: Exactly. Most items get no response, and guesses for them add error. Standard methods score the whole list as one unit, missing which picks helped or hurt, so training gets unstable.

Alex: So the puzzle is fine-tuning each item's role under uncertain data.

Sam: FlexRec fixes that without endless retraining. It creates dynamic suggestions that match changing needs.

Alex: How does it judge each item's role fairly, instead of scoring the whole list?

Sam: Imagine a ranked list, like a lineup of movie suggestions. For one item, they swap it with each item below it, one by one, and check if the list's overall quality—like matching user wants—improves or drops. They average those changes. This "counterfactual swap reward" only looks ahead to unfixed spots, making it causal and fair across lists. Think of it like testing a chess move: does swapping your piece with ones further back strengthen your position locally? That gives each item its own score based on remaining options.

Alex: Does that also handle sparse feedback, or is there more for the noise?

Sam: It creates many signals per list for density. For remaining noise from rare clicks, they upgrade the critic—the part estimating future rewards. It also predicts uncertainty, like a weather forecast's confidence level. Updates then down-weight shaky estimates using inverse variance, stabilizing under sparseness.

FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning

Yijun Pan, Weikang Qiu, Qiyao Ma, Mingxuan Ju, Tong Zhao, Neil Shah, Rex Ying

Mar 12, 2026·5:26·arxiv·Original Paper

LLM-based RecommendersReinforcement Learning (RL) Post-TrainingClosed-Set Autoregressive RankingCausal Item-Level RewardsCritic-Guided Uncertainty-Aware ScalingFlexRec

Listen

5 min

Summary

Abstract

Modern recommender systems must adapt to dynamic, need-specific objectives for diverse recommendation scenarios, yet most traditional recommenders are optimized for a single static target and struggle to reconfigure behavior on demand. Recent advances in reinforcement-learning-based post-training have unlocked strong instruction-following and reasoning capabilities in LLMs, suggesting a principled route for aligning them to complex recommendation goals. Motivated by this, we study closed-set autoregressive ranking, where an LLM generates a permutation over a fixed candidate set conditioned on user context and an explicit need instruction. However, applying RL to this setting faces two key obstacles: (i) sequence-level rewards yield coarse credit assignment that fails to provide fine-grained training signals, and (ii) interaction feedback is sparse and noisy, which together lead to inefficient and unstable updates. We propose FlexRec, a post-training RL framework that addresses both issues with (1) a causally grounded item-level reward based on counterfactual swaps within the remaining candidate pool, and (2) critic-guided, uncertainty-aware scaling that explicitly models reward uncertainty and down-weights low-confidence rewards to stabilize learning under sparse supervision. Across diverse recommendation scenarios and objectives, FlexRec achieves substantial gains: it improves NDCG@5 by up to \textbf{59\%} and Recall@5 by up to \textbf{109.4\%} in need-specific ranking, and further achieves up to \textbf{24.1\%} Recall@5 improvement under generalization settings, outperforming strong traditional recommenders and LLM-based baselines.

FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning: Research Paper Podcast

FlexRec: Adapting LLM-based Recommenders for Flexible Needs via Reinforcement Learning

Listen