MORL-A2C: Multi-Objective Reinforcement Learning Reranker for Optimizing Healthiness in MOPI-HFRS

Traditional food recommendation systems often prioritize user engagement, which can lead to unhealthy dietary patterns. While existing systems like MOPI-HFRS attempt to balance user preference, health, and diversity using Pareto-based optimization, they treat each recommendation step as an independent, static decision. This paper investigates whether reframing food recommendation as a sequential decision-making process—where the value of an item depends on previously selected items—can better navigate the trade-off between user preference and nutritional health.

The authors introduce MORL-A2C, a reinforcement learning extension to the MOPI-HFRS framework. The system uses an Advantage Actor-Critic (A2C) algorithm to learn a policy that constructs a ranked list of recommendations one item at a time. The agent operates on frozen graph neural network (GNN) embeddings, reducing the computational burden of training. To prevent early-training collapse in the large food action space, the policy is warm-started using behavior cloning against a strong dot-product ranker. The agent receives a scalarized reward at each step that combines a relevance signal (based on user interaction history) and a health signal (based on nutritional tag overlap).

By modeling the recommendation process as a sequential Markov Decision Process (MDP), MORL-A2C successfully shifts the system's output toward healthier options. Experimental results show that while the model incurs a modest reduction in standard ranking metrics like Recall@20 (from 25.64% to 23.61%) and NDCG@20 (from 23.52% to 20.64%), it achieves a substantial improvement in health alignment, with the H-Score@20 increasing from 46.05% to 69.57%. The authors also note that they corrected a bug in the original MOPI-HFRS evaluation pipeline, ensuring that these improvements are measured against a fair, corrected baseline.