Rethinking the Suitability of Reinforcement Learning Algorithms Under Practical Transfer Constraints

Standard reinforcement learning (RL) research typically prioritizes sample efficiency—the ability to learn from the fewest possible environment interactions. However, in practical transfer-oriented settings, such as sim-to-real robotics, the cost of generating data is often low compared to the cost of human time and compute resources. This paper argues that researchers and practitioners should shift their focus toward two alternative metrics: wall-clock training time and robustness under dynamics mismatch.

The authors compare three prominent RL paradigms: PPO (on-policy), SAC (off-policy), and TD-MPC2 (model-based). While SAC and TD-MPC2 are generally more sample-efficient, the study demonstrates that PPO often reaches a usable policy faster in terms of wall-clock time. This is because PPO is highly compatible with massively parallel simulation environments, allowing it to collect and process data at a rate that offsets its lower interaction-based efficiency. This finding highlights that the "best" algorithm depends heavily on whether the primary bottleneck is data collection or total training duration.

Domain randomization (DR) is a standard technique used to bridge the reality gap by training agents on a variety of environment dynamics. The study investigates whether different RL paradigms respond differently to this variability. Contrary to the hypothesis that specific paradigms might be inherently more robust, the authors find that PPO, SAC, and TD-MPC2 all benefit from domain randomization in similar ways. The effectiveness of DR is not tied to the algorithm's architecture but is instead highly dependent on the specific task, the coverage of the training distribution, and the evaluation regime.