Automating Potential-based Reward Shaping with Vision Language Model Guidance

Reinforcement learning (RL) agents often struggle in environments with sparse rewards because they lack intermediate feedback to guide exploration. While potential-based reward shaping (PBRS) can provide this guidance without altering the optimal policy, it traditionally requires manual engineering of a potential function. This paper investigates whether vision language models (VLMs) can automatically learn these potential functions from image preferences, thereby reducing human effort and improving sample efficiency.

The authors introduce VLM-PBRS, a framework that uses a lightweight VLM to compare pairs of images and determine which state is closer to a specified goal. These preferences are used to train a potential function, which is then incorporated into the reward signal. Unlike previous methods that use VLMs to define the reward function directly—which can lead to reward hacking—this approach leverages the mathematical properties of PBRS to ensure that the agent's optimal policy remains unchanged, regardless of the potential function's accuracy. By using smaller, more efficient VLMs, the authors minimize the computational cost of repeated queries during training.

Empirical testing in the Meta-World and Franka Kitchen environments demonstrates that VLM-PBRS significantly improves sample efficiency compared to unshaped sparse-reward baselines. The authors show that even when small VLMs produce noisy preference labels, the resulting potential function still provides sufficient guidance to accelerate learning. Furthermore, because the framework is grounded in PBRS, it is robust to reward hacking, as the shaping term cannot incentivize the agent to deviate from the original task objective.