Foundations of Reinforcement Learning and Control: Connections and New Perspectives

Reinforcement learning (RL) and control theory have historically evolved as separate disciplines despite sharing common roots in dynamic programming. Control theory has traditionally focused on stability and robustness in physical systems using Lyapunov functions, while RL has emphasized data-driven optimization and policy learning in uncertain environments. This paper serves as a tutorial to bridge the cultural and methodological gap between these two fields, providing a unified notation and a roadmap for integrating their respective strengths.

The authors highlight that both fields are fundamentally concerned with selecting actions to drive a dynamical system toward a desired goal. While control theory typically models systems using deterministic differential equations and disturbances, RL uses stochastic Markov decision processes. The paper demonstrates that these perspectives are complementary: Lyapunov functions, which certify stability in control, share a deep mathematical connection with the value functions used in RL to quantify long-term performance. By aligning these notations, the authors show how modern data-driven control and model-free RL can be viewed as different approaches to solving the same underlying optimization problem.

To demonstrate the value of this synthesis, the authors introduce a hybrid control architecture tested on the Half-Cheetah robotic locomotion task. They combine an actor-critic RL algorithm—which excels at learning complex, high-level behaviors—with Model-Reference Adaptive Control (MRAC), which provides robust, real-time adaptation for low-level joint stabilization. This combination allows the system to achieve high performance in task execution while maintaining the stability guarantees required for physical hardware, suggesting a powerful template for future robotics applications.