Mesh-RL: Coupled subgrid reinforcement learning

Reinforcement learning (RL) agents often struggle in large or sparse-reward environments because value information propagates slowly through the state space. Standard temporal-difference (TD) methods rely on local updates that take many episodes to reach distant states. This paper investigates whether spatial domain decomposition—a technique commonly used in scientific computing to solve complex partial differential equations—can be adapted to accelerate reward propagation in RL without requiring complex hierarchical structures or model-based planning.

The authors introduce Mesh-RL, a framework that divides the environment into overlapping subgrids. Each subgrid acts as an independent learning unit that performs standard TD updates. To ensure global coherence, the framework enforces boundary-consistent updates: when an agent transitions across a subgrid boundary, the TD target incorporates value estimates from the adjacent subgrid. After each episode, the local Q-values are stitched together by averaging overlapping regions, ensuring that information about the goal propagates upstream more efficiently than in a monolithic state space.

Mesh-RL consistently improves convergence speed, cumulative reward, and learning stability across Q-learning, SARSA, and Dyna-Q. By increasing the number of mesh partitions, the authors demonstrate that the agent sustains exploration longer and avoids premature convergence to suboptimal policies. The results show that even for algorithms like Dyna-Q, which already utilize internal planning, the structured decomposition provided by Mesh-RL offers additional performance gains. Heatmap visualizations confirm that Mesh-RL produces smoother, more globally consistent value functions compared to standard baseline methods.