Learning Process Rewards via Success Visitation Matching for Efficient RL

Reinforcement learning (RL) often relies on sparse outcome rewards—where an agent receives a reward only upon task completion. This sparsity creates a difficult credit assignment problem, as the agent struggles to determine which specific actions led to success. The authors investigate how to automatically generate dense 'process rewards' that provide step-level feedback, thereby accelerating learning without requiring manual reward engineering or expert demonstrations.

The proposed Success Visitation Matching (SVM) method uses a discriminator to compare state-action visitations from successful episodes against those from unsuccessful ones. By calculating the log-probability ratio of these visitations, the system generates a dense reward signal that incentivizes the agent to mimic successful behaviors and avoid unsuccessful ones. This process is integrated into standard RL pipelines by alternating between updating the discriminator with new experience and training the policy to maximize this shaped reward.

The authors prove that in deterministic environments, any policy maximizing the SVM reward is also optimal for the original sparse outcome reward. In practice, they demonstrate that SVM significantly accelerates the finetuning of robotic control policies in both simulation and real-world manipulation tasks. Compared to baseline methods that rely solely on sparse rewards, SVM enables faster and more stable convergence, often succeeding in tasks where sparse-reward RL fails to learn entirely.

This work provides a scalable, automated solution to the reward sparsity problem in robotics. By leveraging only the sparse outcome signal already present in many RL environments, SVM avoids the need for expensive human-in-the-loop feedback or hand-crafted reward functions, making it a highly practical tool for improving the sample efficiency of robotic learning systems.