Memory-Efficient Policy Libraries with Low-Rank Adaptation in Reinforcement Learning

Research Question and Approach

This paper investigates whether Low-Rank Adaptation (LoRA)—a parameter-efficient fine-tuning technique widely used in Large Language Models—can be effectively applied to multi-task reinforcement learning (RL) in robotics. In robotics, creating a library of specialized policies for different tasks often leads to high memory and storage costs due to catastrophic forgetting, which prevents a single policy from easily mastering many distinct tasks. The authors propose using LoRA to create a library of specialist policies by freezing a pre-trained base policy and training only small, low-rank matrices for each new task. They evaluate this approach using the Proximal Policy Optimization (PPO) algorithm on the Meta-World benchmark suite.

The researchers found that LoRA significantly reduces the number of trainable parameters, achieving a 20-160x reduction in memory usage compared to full fine-tuning. This translates to 90-95% storage savings when maintaining a library of 10-50 specialized policies. Crucially, the study demonstrates that these memory gains do not come at the cost of performance; for the selected tasks, LoRA-based policies achieved success rates comparable to those trained via full fine-tuning. The authors also observed that while simpler tasks can be solved with a very low LoRA rank (e.g., rank 1), more complex tasks may require higher ranks (e.g., 2-8) to reach optimal performance.

In applied robotics, the ability to store a large library of specialized policies in memory is often the difference between seamless task switching and the need for slow, inefficient swap-memory usage. By demonstrating that LoRA is a viable, memory-efficient alternative to full fine-tuning in RL, this work provides a practical pathway for deploying versatile, multi-task robotic systems that can adapt to new environments without the prohibitive storage overhead of traditional methods.