Generalize and Guide: Decomposing Rewards for Few-Shot Inverse Reinforcement Learning

How can an agent learn a new task with substantial natural variations (e.g., different object configurations or layouts) when only a limited number of target-task demonstrations are available? The authors address this by proposing a new problem setting called Few-Shot IRL with Multi-Task Demonstrations (FM-IRL).

The authors introduce MPG, a reward decomposition framework that combines two complementary signals to guide policy learning:

1. Generalizable Discriminator: This component leverages a large dataset of related task demonstrations to learn shared structure. It acts as a classifier to identify expert-consistent behavior, allowing the agent to transfer knowledge from related tasks to the target task.
2. Proximity Function: This component provides dense, corrective feedback during online exploration. It estimates the temporal distance (in steps) from the agent's current state to the expert state distribution, effectively guiding the agent back to expert behavior when it deviates.

The final reward function is a combination of the discriminator's expert-recognition score and the proximity function's improvement signal, which is then used to optimize the agent's policy via reinforcement learning.

MPG was evaluated across diverse navigation and manipulation domains, including maze navigation and block stacking. The method achieved an average success rate of 81.2%, outperforming the strongest per-task baseline by an average of 24.7 percentage points. The results suggest that combining multi-task knowledge transfer with online proximity-based guidance is highly effective for sample-efficient learning in environments with significant intra-task variations.

Traditional inverse reinforcement learning often struggles with natural variations because collecting enough demonstrations to cover every scenario is prohibitively expensive. By demonstrating that agents can effectively "practice" and refine their behavior using related-task data and online interaction, this work provides a scalable path toward deploying robots in complex, real-world environments where full expert coverage is impossible.