VGFM: Expressive Robot Policies via Dense Value Guidance in Flow Matching

Offline reinforcement learning (RL) often struggles with distribution shift and the need for expressive, multimodal action representations. While generative models like diffusion and flow matching offer a solution, integrating them into value-based actor-critic frameworks typically requires backpropagation through time (BPTT) over the entire generative trajectory. This paper asks whether it is possible to achieve dense value-guided policy improvement in flow-based models while avoiding the computational overhead and instability associated with BPTT.

The authors propose Value-Guided Flow Matching (VGFM), a framework that parameterizes the policy as a conditional flow-matching model in action (x-prediction) space. By predicting the terminal action at each intermediate flow step, the model ensures that every step produces a valid action that can be evaluated by a standard offline RL critic. This design allows the agent to apply value-based supervision at any sampled flow time without needing to differentiate through the entire generative process. The training objective combines a flow-matching loss for behavior regularization with a value-guidance loss for policy improvement, effectively steering the velocity field toward higher-value actions.

VGFM demonstrates strong performance across a wide range of robotic locomotion and manipulation tasks in the OGBench benchmark. By decoupling the value-guidance signal from the full generative trajectory, the model achieves stable training and inference-time scalability. Users can adjust the number of discretization steps during inference to trade off compute for control precision without requiring retraining. The authors show that VGFM matches or outperforms established baselines, including Gaussian, diffusion, and other flow-based policies, while maintaining a simpler algorithmic structure that avoids auxiliary losses or BPTT.