Hongru Hou, Tiehua Mei, Denghui Geng, Jinhui Huang, Ao Xu, Hengrui Chen, Jiaqing Liang, Deqing Yang
7 min
Abstract
Proactive Recommender Systems (PRSs) aim to guide user preference shift toward target items by generating paths of intermediate recommendations. Reinforcement learning (RL) provides a principled framework for optimizing such sequential decision tasks, as path rewards can naturally capture both short-term acceptance and long-term guidance effectiveness. However, naively applying policy gradients to PRS results in deficient gradient estimation. We identify two deficiencies: (1) path-level rewards decompose into step-level rewards with positive mean, creating a length-dependent bias that causes gradients to favor path extension over meaningful exploration; (2) weighting each step by the entire path-level reward ignores the decomposition structure, leading to high gradient variance. To rectify these two deficiencies, we propose an effective RL framework ProRL with two novel mechanisms for proactive recommendation. First, Stepwise Reward Centering subtracts expected rewards to neutralize length-dependent bias, ensuring that path extension yields zero expected gradient signal. Second, Position-Specific Advantage Estimation leverages the reward decomposition structure to compute step-dependent baselines, reducing gradient variance. Together, these mechanisms yield policy gradients that precisely target path quality. Our experiments on three real-world datasets demonstrate that ProRL significantly outperforms state-of-the-art PRSs. Our code is available at https://github.com/hongruhou89/ProRL.
Alex: So how do you stop it gaming the system like that?
Sam: The fix is called Stepwise Reward Centering. Instead of handing out a point just for making a suggestion the user might accept, the system first asks: "How good is this step compared to what we'd normally expect at this point in the journey?" It calculates the average reward the model usually earns at each position, and then subtracts that average from the actual score.
Alex: So if "average" is now worth zero, the only way to earn a positive reward is to do something genuinely better than usual.
Sam: Right. A mediocre step earns nothing. A shortcut earns nothing. The only thing that gets rewarded is a suggestion that's meaningfully better than the baseline. That single change removes the incentive to pad the path.
Alex: But I imagine knowing *which* step in a long chain actually made the difference is still tricky?
Sam: That's the second problem. When the computer is learning, it tends to look at the entire path and try to figure out what went right or wrong all at once. It's like trying to work out which specific move lost you a game of chess by only looking at the final score. You lose all the detail.
Alex: So how do you fix that?
Sam: The researchers developed what they call Position-Specific Advantage Estimation. The idea is that instead of judging a step by the overall outcome, you judge it only by what happens *after* that step — and you compare it to how well things typically go from that exact position in the sequence. Think of it like judging a marathon runner not by their total finish time, but by how they ran the third mile compared to how everyone else typically runs that specific mile. You get a much more precise picture of what actually helped.
Alex: So every plank on the bridge gets its own individual grade, based on how much closer it moved you to the other side.
Sam: Precisely. And for this to work at all, the model also needs a clear sense of how items relate to each other. The researchers gave it what you might call a map — a way of representing every item, whether a film or a book, as a kind of code that captures its meaning and genre. They call these Semantic IDs. The model uses this map to understand that a space movie is closer to a robot movie than it is to a cooking show.
Alex: That makes sense. So it's not just memorising which items are popular — it actually understands the territory.
Sam: And the training happens in two stages. First, the model learns the "language" of recommendations by studying thousands of real examples where users naturally moved from one interest to another. Then, the reward system we described fine-tunes its strategy — pushing it to actually reach the target, not just wander nearby.
Alex: How do you measure whether it's actually working? Couldn't it just be recommending the same popular items to everyone?
Sam: That's exactly what the researchers tested for. Their key measure was how much the target item moved up in a user's personal preference ranking after they'd seen the recommended path. In their tests, the target item became significantly more prominent — moving up roughly twice as much compared to older methods. They also measured what they call "Coherence" — whether the items in the chain actually make logical sense together. A path from hammer to screwdriver to drill is coherent. A path from hammer to a block of cheese is not.
Alex: And ProRL kept the paths coherent while still reaching the target?
Sam: It did. And what's particularly worth noting is that the model already had the underlying knowledge to find these paths from its initial training. It just needed the right reward structure to actually *choose* them. By removing the incentive to take shortcuts, the researchers turned a system that was essentially lazy into one that could plan a meaningful journey.
Alex: Though I have to wonder — if the computer is learning from a simulated model of a person rather than a real human, couldn't it pick up the wrong lessons?
Sam: That's a real limitation the researchers acknowledge. The reward signals come from a mathematical model of user behaviour. If that model is biased, or doesn't perfectly capture how a real person responds to a sequence of suggestions, the paths the system generates might not work as well outside the lab. The bridge it builds is only as good as its guess of where the person actually wants to walk.
Alex: So the quality of the guidance is capped by how well we can predict human interest.
Sam: Exactly. That said, the approach held up across different types of data — from films to books — which suggests the core logic is reasonably robust. And the near-term implications go beyond shopping or streaming. A system like this could act as a teaching tool, gradually expanding someone's skills or knowledge through a sequence that feels natural rather than forced. It moves away from "more of the same" and toward something closer to genuine discovery.
Alex: A more thoughtful way to navigate the digital world. Thanks for walking us through it, Sam. And thanks for listening to ResearchPod.